Overview

Clustering is an unsupervised learning method whose task is to divide the population or data points into a number of groups, such that data points in a group are more similar to other data points in the same group and dissimilar to the data points in other groups. It is basically a collection of objects based on similarity and dissimilarity between them.

K-means clustering is a type of unsupervised learning, which is used when you have unlabeled data (i.e., data without defined categories or groups). The goal of this algorithm is to find groups in the data, with the number of groups represented by the variable K. Given the fact that the dataset provided for Categorical Feature Encoding have 100% categorical variables, we are performing the kmeans clustering analysis by employing two different encoding techniques, viz, one hot encoding and similarity encoding of the image_name feature.

In this notebook, we explore the distribution of image_name repos for differenct cluster_id based on its frequency of its occurance in a cluster_id and also based on excluding the frequency of occurance in a cluster_id. We approach clustering, by two different encoding techniques, i.e, One Hot Encoding and Similarity Encoding.

Importing Packages

import io
import boto3
import pandas as pd
import numpy as np
import warnings
import os
import matplotlib.pyplot as plt
import plotly.express as px

from dirty_cat import SimilarityEncoder
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from tqdm import tqdm

from dotenv import load_dotenv, find_dotenv
%matplotlib inline
load_dotenv(find_dotenv())
warnings.filterwarnings("ignore")

pd.set_option("display.max_rows", None)
pd.set_option("display.max_columns", None)
# CEPH Bucket variables
s3_endpoint_url = os.getenv("S3_ENDPOINT")
s3_access_key = os.getenv("S3_ACCESS_KEY")
s3_secret_key = os.getenv("S3_SECRET_KEY")
s3_bucket = os.getenv("S3_BUCKET")

# s3 resource to communicate with storage
s3 = boto3.resource(
    "s3",
    endpoint_url=s3_endpoint_url,
    aws_access_key_id=s3_access_key,
    aws_secret_access_key=s3_secret_key,
)

# access the parquet file as an s3 object

obj1 = s3.Object(
    "DH-PLAYPEN", "ccx/fingerprinting/image_layers/date=2021-05-12/2021-05-12.parquet"
)
obj2 = s3.Object(
    "DH-PLAYPEN", "ccx/fingerprinting/containers/date=2021-05-12/2021-05-12.parquet"
)
obj3 = s3.Object(
    "DH-PLAYPEN", "ccx/fingerprinting/image_layers/dataframe_image_id.parquet"
)
obj4 = s3.Object("DH-PLAYPEN", "ccx/fingerprinting/containers/df_cont_image_id.parquet")
# download the file into the buffer
buffer1 = io.BytesIO()
obj1.download_fileobj(buffer1)
buffer2 = io.BytesIO()
obj2.download_fileobj(buffer2)
buffer3 = io.BytesIO()
obj3.download_fileobj(buffer3)
buffer4 = io.BytesIO()
obj4.download_fileobj(buffer4)

# read the buffer and create the dataframe
image_layers_df = pd.read_parquet(buffer1)
image_id_map = pd.read_parquet(buffer3)
containers_df = pd.read_parquet(buffer2)
cont_id_map = pd.read_parquet(buffer4)

We now have the four sets of dataframe,

  • Image layer dataset (image_layers_df)

  • Product name mapping for sha’s of image_id from image layer dataset (image_id_mapping)

  • Containers dataset (containers_df)

  • Product name mapping for sha’s of the image_id from container dataset (cont_imageid_map)

Image layer Dataset

image_layers_df.head()
cluster_id image_id layer_image_id layer_image_level first_command first_arg archive_path
0 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... sha256:9ebb302e1fb002fb643091710dac46f8258781d... 0 icTsn2s_EIax 2v1NneeWoS_9 archives/compressed/00/00003d61-9db1-4757-9cd1...
1 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... sha256:a74396a32e85c2feeedf76052ed3297859810c8... 1 icTsn2s_EIax 2v1NneeWoS_9 archives/compressed/00/00003d61-9db1-4757-9cd1...
2 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... sha256:7db62383a7470afbacfc0fab55d5a182e3c5fa2... 2 icTsn2s_EIax 2v1NneeWoS_9 archives/compressed/00/00003d61-9db1-4757-9cd1...
3 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... sha256:f24250419f728db23957454976d6d38b679a349... 3 icTsn2s_EIax 2v1NneeWoS_9 archives/compressed/00/00003d61-9db1-4757-9cd1...
4 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... sha256:267f7bb0f5dcf1b83f8ce89831d05f3a44a3abe... 4 icTsn2s_EIax 2v1NneeWoS_9 archives/compressed/00/00003d61-9db1-4757-9cd1...
image_id_map.head()
License architecture build-date com.redhat.build-host com.redhat.component com.redhat.license_terms description distribution-scope io.k8s.description io.k8s.display-name io.openshift.build.commit.id io.openshift.build.commit.url io.openshift.build.source-location io.openshift.expose-services io.openshift.maintainer.component io.openshift.maintainer.product io.openshift.tags maintainer name release summary url vcs-ref vcs-type vendor version io.openshift.s2i.scripts-url io.s2i.scripts-url usage io.openshift.s2i.assemble-user authoritative-source-url io.fabric8.s2i.version.jolokia org.concrt.version org.jboss.product org.jboss.product.amq.version org.jboss.product.openjdk.version org.jboss.product.version com.redhat.deployments-dir com.redhat.dev-mode io.jenkins.version
image_id
sha256:337c22cabe530213b14965f9ea69a92dbeb5104f79b00fd7861db7057a9e05ae GPLv2+ x86_64 2021-04-30T00:31:42.349887 cpt-1004.osbs.prod.upshift.rdu2.redhat.com ose-cli-artifacts-container https://www.redhat.com/agreements OpenShift is a platform for developing, buildi... public OpenShift is a platform for developing, buildi... OpenShift Clients a765590e1b87b014b9d81f9ea534460d6dff73f2 https://github.com/openshift/oc/commit/a765590... https://github.com/openshift/oc oc OpenShift Container Platform openshift,cli Red Hat, Inc. openshift/ose-cli-artifacts 202104292348.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... 43f412b2932a5ecfe5ebedcab38cf2915cf03813 git Red Hat, Inc. v4.8.0 None None None None None None None None None None None None None None
sha256:3574d6c1fcc46e1ebd41b7b887b92035ea18213133c06edad539cde25b767ff9 GPLv2+ x86_64 2021-04-30T00:22:29.539443 cpt-1008.osbs.prod.upshift.rdu2.redhat.com openshift-enterprise-cli-container https://www.redhat.com/agreements OpenShift is a platform for developing, buildi... public OpenShift is a platform for developing, buildi... OpenShift Client a765590e1b87b014b9d81f9ea534460d6dff73f2 https://github.com/openshift/oc/commit/a765590... https://github.com/openshift/oc oc OpenShift Container Platform openshift,cli Red Hat, Inc. openshift/ose-cli 202104292348.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... d3ae20f4e2fac18ea671cf12636d16791146a460 git Red Hat, Inc. v4.8.0 None None None None None None None None None None None None None None
sha256:c588b4653ef037b1300f9692b986cbeac5dfe5e66f6605ca13e0565202e8fe02 GPLv2+ x86_64 2021-05-04T21:53:04.228477 cpt-1005.osbs.prod.upshift.rdu2.redhat.com golang-github-openshift-oauth-proxy-container https://www.redhat.com/agreements OpenShift OAuth Proxy. public OpenShift OAuth Proxy. OpenShift OAuth Proxy 4bd4705011c19a7556551c9a1dfaaa5eacc7898c https://github.com/openshift/oauth-proxy/commi... https://github.com/openshift/oauth-proxy Release OpenShift Container Platform oauth Red Hat, Inc. openshift/ose-oauth-proxy 202105042126.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... 6f8aceb12953fba6382c52a2a99200bcab6c6f6b git Red Hat, Inc. v4.8.0 None None None None None None None None None None None None None None
sha256:7b879d0179b96fe682a186af60836e4091f1fd3148d4cc5e9b10b3fc65bd2900 GPLv2+ x86_64 2021-04-29T23:09:36.775464 cpt-1008.osbs.prod.upshift.rdu2.redhat.com golang-github-openshift-oauth-proxy-container https://www.redhat.com/agreements OpenShift OAuth Proxy. public OpenShift OAuth Proxy. OpenShift OAuth Proxy 4bd4705011c19a7556551c9a1dfaaa5eacc7898c https://github.com/openshift/oauth-proxy/commi... https://github.com/openshift/oauth-proxy Release OpenShift Container Platform oauth Red Hat, Inc. openshift/ose-oauth-proxy 202104292205.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... 1bbf5dd3c77a259d830eb5182c2f5cf0f6431a20 git Red Hat, Inc. v4.8.0 None None None None None None None None None None None None None None
sha256:85a79d95924e4a76b1a6ccd974b2c99df349f0743ebb515c0ffcd00ef107c84c GPLv2+ x86_64 2021-05-08T07:13:14.635470 cpt-1007.osbs.prod.upshift.rdu2.redhat.com golang-github-openshift-oauth-proxy-container https://www.redhat.com/agreements OpenShift OAuth Proxy. public OpenShift OAuth Proxy. OpenShift OAuth Proxy 17a8d9f5180a5838fa253c277bc4c6fa71c4fd1e https://github.com/openshift/oauth-proxy/commi... https://github.com/openshift/oauth-proxy Release OpenShift Container Platform oauth Red Hat, Inc. openshift/ose-oauth-proxy 202105080658.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... 5d6faed434c1fe7855eed599bfa66764b5615eee git Red Hat, Inc. v4.8.0 None None None None None None None None None None None None None None

We include only name column from the mapped dataset for further analysis.

image_id_map = image_id_map.reset_index()
image_id_map = image_id_map[["image_id", "name"]]
image_id_map.head()
image_id name
0 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... openshift/ose-cli-artifacts
1 sha256:3574d6c1fcc46e1ebd41b7b887b92035ea18213... openshift/ose-cli
2 sha256:c588b4653ef037b1300f9692b986cbeac5dfe5e... openshift/ose-oauth-proxy
3 sha256:7b879d0179b96fe682a186af60836e4091f1fd3... openshift/ose-oauth-proxy
4 sha256:85a79d95924e4a76b1a6ccd974b2c99df349f07... openshift/ose-oauth-proxy

Container dataset

containers_df.head()
cluster_id namespace shape shape_instances image_id first_command first_arg init_container archive_path
0 00003d61-9db1-4757-9cd1-84df271daeb9 0LiT6ZNtbpYL sha256:3ecf29979b2722bf4a82a5e7a954e8685820720... 1 sha256:f46f210d6023bec16e68340b484a8881ce46d5e... None 47DEQpj8HBSa False archives/compressed/00/00003d61-9db1-4757-9cd1...
1 00003d61-9db1-4757-9cd1-84df271daeb9 0LiT6ZNtbpYL sha256:3ecf29979b2722bf4a82a5e7a954e8685820720... 1 sha256:edb9aaacf421c6dc45b20324e8699cec02f26bf... n9CdwzVF-cwZ RNOaw_AuQeIY False archives/compressed/00/00003d61-9db1-4757-9cd1...
2 00003d61-9db1-4757-9cd1-84df271daeb9 0LiT6ZNtbpYL sha256:542d007d13008cc1be2dbf03601b954c4452947... 1 sha256:a693c315b775c693dc49c19b7f217762676bc28... b51B0EZ1bw3c ua-xlwwsvdYd False archives/compressed/00/00003d61-9db1-4757-9cd1...
3 00003d61-9db1-4757-9cd1-84df271daeb9 0LiT6ZNtbpYL sha256:542d007d13008cc1be2dbf03601b954c4452947... 1 sha256:a693c315b775c693dc49c19b7f217762676bc28... Cl6kTzfbYztA None True archives/compressed/00/00003d61-9db1-4757-9cd1...
4 00003d61-9db1-4757-9cd1-84df271daeb9 0LiT6ZNtbpYL sha256:542d007d13008cc1be2dbf03601b954c4452947... 1 sha256:d9c64d038f16e04c52142bc9e7dfa0645ce7e34... Cl6kTzfbYztA None True archives/compressed/00/00003d61-9db1-4757-9cd1...
cont_id_map.head()
License architecture build-date com.redhat.build-host com.redhat.component com.redhat.license_terms description distribution-scope io.k8s.description io.k8s.display-name io.openshift.build.commit.id io.openshift.build.commit.url io.openshift.build.source-location io.openshift.expose-services io.openshift.maintainer.component io.openshift.maintainer.product io.openshift.tags maintainer name release summary url vcs-ref vcs-type vendor version io.openshift.maintainer.subcomponent io.openshift.release.operator io.openshift.build.versions com.redhat.delivery.appregistry upstream-vcs-ref upstream-vcs-type upstream-version org.kubevirt.hco.csv-generator.v1 io.openshift.s2i.scripts-url io.s2i.scripts-url usage io.openshift.s2i.assemble-user display-name com.redhat.delivery.operator.bundle com.redhat.openshift.versions io.cekit.version operators.operatorframework.io.bundle.channel.default.v1 operators.operatorframework.io.bundle.channels.v1 operators.operatorframework.io.bundle.manifests.v1 operators.operatorframework.io.bundle.mediatype.v1 operators.operatorframework.io.bundle.metadata.v1 operators.operatorframework.io.bundle.package.v1 licenses CEPH_POINT_RELEASE GIT_BRANCH GIT_CLEAN GIT_COMMIT GIT_REPO RELEASE ocs.tags com.redhat.deployments-dir com.redhat.dev-mode com.redhat.dev-mode.port help operators.operatorframework.io.index.database.v1 authoritative-source-url license io.fabric8.s2i.version.jolokia io.fabric8.s2i.version.maven io.openshift.s2i.destination org.jboss.container.deployments-dir org.jboss.product org.jboss.product.eap.version org.jboss.product.openjdk.version org.jboss.product.sso.version org.jboss.product.version istio_version openshift_build operator_build run org.concrt.version org.jboss.product.amq.version io.openshift.build.commit.author io.openshift.build.commit.date io.openshift.build.commit.message io.openshift.build.commit.ref io.openshift.build.name io.openshift.build.namespace io.openshift.build.source-context-dir jenkins.build.number jenkins.tarball.url io.jenkins.version build-utility org.label-schema.vcs-ref org.label-schema.vcs-url org.label-schema.description org.label-schema.license org.label-schema.name org.label-schema.schema-version org.label-schema.vendor JAVA_VERSION com.ibm.events.commitid com.ibm.eventstreams.base-for-bedrock.icp-linux-amd64.commitid com.ibm.eventstreams.base-for-bedrock.icp-linux-amd64.job com.ibm.eventstreams.base-for-bedrock.icp-linux-amd64.license com.ibm.eventstreams.base-for-bedrock.icp-linux-amd64.maintainer com.ibm.eventstreams.base-for-bedrock.icp-linux-amd64.name com.ibm.eventstreams.base-for-bedrock.icp-linux-amd64.version com.ibm.eventstreams.openjdk-11-sdk-for-bedrock.icp-linux-amd64.commitid com.ibm.eventstreams.openjdk-11-sdk-for-bedrock.icp-linux-amd64.job com.ibm.eventstreams.openjdk-11-sdk-for-bedrock.icp-linux-amd64.license com.ibm.eventstreams.openjdk-11-sdk-for-bedrock.icp-linux-amd64.maintainer com.ibm.eventstreams.openjdk-11-sdk-for-bedrock.icp-linux-amd64.name com.ibm.eventstreams.openjdk-11-sdk-for-bedrock.icp-linux-amd64.version com.redhat.apb.runtime com.microsoft.product com.microsoft.version
image_id
sha256:f46f210d6023bec16e68340b484a8881ce46d5e6749f7c7f86f9840709d2e080 ASL 2.0 x86_64 2021-05-04T21:54:58.392948 cpt-1007.osbs.prod.upshift.rdu2.redhat.com kube-rbac-proxy-container https://www.redhat.com/agreements This is a proxy, that can perform Kubernetes R... public This is a proxy, that can perform Kubernetes R... kube-rbac-proxy 8d11a8fa9ce252cd25794c0d9280cbdc0c2affcb https://github.com/openshift/kube-rbac-proxy/c... https://github.com/openshift/kube-rbac-proxy Monitoring OpenShift Container Platform kubernetes OpenShift Monitoring Team <team-monitoring@red... openshift/ose-kube-rbac-proxy 202105042126.p0 https://access.redhat.com/containers/#/registr... 12ef9d3cc226f6bd4a898d4b23ffa1ec5d3d27f1 git Red Hat, Inc. v4.8.0 None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None
sha256:edb9aaacf421c6dc45b20324e8699cec02f26bf34749b72fd729fe5cdaf78133 GPLv2+ x86_64 2021-05-04T22:05:48.018566 cpt-1006.osbs.prod.upshift.rdu2.redhat.com ose-multus-admission-controller-container https://www.redhat.com/agreements This is a component of OpenShift Container Pla... public This is a component of OpenShift Container Pla... Container Networking Plugins a7312f5e55e9f34cc8b20f6cbfe1af0f363ca1e6 https://github.com/openshift/multus-admission-... https://github.com/openshift/multus-admission-... Networking OpenShift Container Platform openshift Doug Smith <dosmith@redhat.com> openshift/ose-multus-admission-controller 202105042126.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... 00692865fc2dd0c845bb20c688dbf2cb7e239062 git Red Hat, Inc. v4.8.0 multus None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None
sha256:a693c315b775c693dc49c19b7f217762676bc287d2f9c5f635e1e1bb23124232 GPLv2+ x86_64 2021-05-04T21:53:09.591552 cpt-1006.osbs.prod.upshift.rdu2.redhat.com multus-cni-container https://www.redhat.com/agreements This is a component of OpenShift Container Pla... public This is a component of OpenShift Container Pla... Multus CNI 4839889c619472ad0ef6868a0eeb2685363ede60 https://github.com/openshift/multus-cni/commit... https://github.com/openshift/multus-cni Networking OpenShift Container Platform openshift Doug Smith <dosmith@redhat.com> openshift/ose-multus-cni 202105042126.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... dfd382c4f51a27e76aa8c5888d7371b379f3376a git Red Hat, Inc. v4.8.0 multus None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None
sha256:d9c64d038f16e04c52142bc9e7dfa0645ce7e3481c5cf4ae819a1411b8aafc7f GPLv2+ x86_64 2021-05-04T22:19:44.056847 cpt-1003.osbs.prod.upshift.rdu2.redhat.com ose-egress-router-cni-container https://www.redhat.com/agreements CNI Plugin for Egress Router public CNI Plugin for Egress Router Egress Router CNI a0909e3a357d2f977ffd80d84e0e0e22afc32ae0 https://github.com/openshift/egress-router-cni... https://github.com/openshift/egress-router-cni Networking OpenShift Container Platform openshift Daniel Mellado <dmellado@redhat.com> openshift/ose-egress-router-cni 202105042126.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... f9b7a21d586e42555c9922a76beff0a4775d2482 git Red Hat, Inc. v4.8.0 None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None
sha256:af32a5e5489c79461bf64c6e55bd34e1c1c1bd941e8cc6381f9124349a492054 GPLv2+ x86_64 2021-05-04T22:18:44.217005 cpt-1004.osbs.prod.upshift.rdu2.redhat.com ose-containernetworking-plugins-container https://www.redhat.com/agreements This is a component of OpenShift Container Pla... public This is a component of OpenShift Container Pla... Container Networking Plugins a8801b05722b803db020506231f645f93cf36e7d https://github.com/openshift/containernetworki... https://github.com/openshift/containernetworki... Networking OpenShift Container Platform openshift Doug Smith <dosmith@redhat.com> openshift/ose-container-networking-plugins 202105042126.p0 Provides the latest release of Red Hat Univers... https://access.redhat.com/containers/#/registr... a276e35519240cca86edaa17231e9a8dff146996 git Red Hat, Inc. v4.8.0 multus None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None
cont_id_map = cont_id_map.reset_index()
cont_id_map = cont_id_map[["image_id", "name"]]
cont_id_map.head()
image_id name
0 sha256:f46f210d6023bec16e68340b484a8881ce46d5e... openshift/ose-kube-rbac-proxy
1 sha256:edb9aaacf421c6dc45b20324e8699cec02f26bf... openshift/ose-multus-admission-controller
2 sha256:a693c315b775c693dc49c19b7f217762676bc28... openshift/ose-multus-cni
3 sha256:d9c64d038f16e04c52142bc9e7dfa0645ce7e34... openshift/ose-egress-router-cni
4 sha256:af32a5e5489c79461bf64c6e55bd34e1c1c1bd9... openshift/ose-container-networking-plugins

Using the mapping techniques from the issue, we were able to map most (not all) of the image_id from the image layers dataset and containers datsets. In the next steps, we try to merge the two datset with respect to the image_id. The ‘product name’ and ‘summary’ of the image_id that were not mapped were left blank.

In the next section, we merge the two dataset.

Merging the datasets with respect to the image_id

Merged dataframe for image layers dataset

df_image = pd.merge(image_layers_df, image_id_map, on="image_id", how="outer")
df_image = df_image[["cluster_id", "image_id", "name"]]
df_image.head()
cluster_id image_id name
0 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... openshift/ose-cli-artifacts
1 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... openshift/ose-cli-artifacts
2 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... openshift/ose-cli-artifacts
3 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... openshift/ose-cli-artifacts
4 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:337c22cabe530213b14965f9ea69a92dbeb5104... openshift/ose-cli-artifacts

Merged dataframe for containers dataset

df_cont = pd.merge(containers_df, cont_id_map, on="image_id", how="outer")
df_cont = df_cont[["cluster_id", "image_id", "name"]]
df_cont.head()
cluster_id image_id name
0 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:f46f210d6023bec16e68340b484a8881ce46d5e... openshift/ose-kube-rbac-proxy
1 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:f46f210d6023bec16e68340b484a8881ce46d5e... openshift/ose-kube-rbac-proxy
2 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:f46f210d6023bec16e68340b484a8881ce46d5e... openshift/ose-kube-rbac-proxy
3 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:f46f210d6023bec16e68340b484a8881ce46d5e... openshift/ose-kube-rbac-proxy
4 00003d61-9db1-4757-9cd1-84df271daeb9 sha256:f46f210d6023bec16e68340b484a8881ce46d5e... openshift/ose-kube-rbac-proxy

Replacing the nan values for name with string Not Identified

We observed that there are image_id’s which could not be mapped and are therefore represented as nan values. We drop those rows containing the nan values for name column.

# df_image.name.fillna('Not Identified', inplace=True)
# df_cont.name.fillna('Not Identified', inplace=True)
df_image = df_image[df_image.name.notnull()]
df_cont = df_cont[df_cont.name.notnull()]
df_image.tail()
cluster_id image_id name
47311 d3c907ae-df45-49ee-8a62-0ac4d5f6e295 sha256:57bbcd1b0082f315fba4e11d2abd1aa744db1b9... openshift/ose-jenkins
47312 d3c907ae-df45-49ee-8a62-0ac4d5f6e295 sha256:57bbcd1b0082f315fba4e11d2abd1aa744db1b9... openshift/ose-jenkins
47313 d3c907ae-df45-49ee-8a62-0ac4d5f6e295 sha256:57bbcd1b0082f315fba4e11d2abd1aa744db1b9... openshift/ose-jenkins
47314 d3c907ae-df45-49ee-8a62-0ac4d5f6e295 sha256:57bbcd1b0082f315fba4e11d2abd1aa744db1b9... openshift/ose-jenkins
47315 d3c907ae-df45-49ee-8a62-0ac4d5f6e295 sha256:57bbcd1b0082f315fba4e11d2abd1aa744db1b9... openshift/ose-jenkins
df_image = df_image.reset_index()
df_image = df_image[["cluster_id", "image_id", "name"]]
df_cont = df_cont.reset_index()
df_cont = df_cont[["cluster_id", "image_id", "name"]]

In the next section, we will print the number of unique categories per column to help us understand the data.

For image_layers dataset

for col in df_image.columns:
    print(f"Number of unique categories in {col} = {df_image[col].nunique()}\n")
Number of unique categories in cluster_id = 1810

Number of unique categories in image_id = 111

Number of unique categories in name = 25

For container dataset

for col in df_cont.columns:
    print(f"Number of unique categories in {col} = {df_cont[col].nunique()}\n")
Number of unique categories in cluster_id = 1881

Number of unique categories in image_id = 2478

Number of unique categories in name = 353

Encoding categorical features

For encoding the categorical feature, we try to employ two different types of encoding techniques, viz, One Hot Encoding and Similarity Encoding. We are clustering the dataset based on cluster_id and image_id for bith image_layer dataset and container dataset.

One Hot Encoding

We have 25 unique name and 1810 cluster_id for image_layer_dataset and 353 different names and 1881 different cluster_id for container dataset. After one hot encoding, we perform Kmeans clustering for both the datasets.

Image_layer_dataset

df_image_name = df_image[["cluster_id", "name"]]
dummy = pd.get_dummies(df_image_name.name)
new_image = pd.concat([df_image_name, dummy], axis=1)
new_image = new_image.drop(columns="name")
new_image = new_image.set_index("cluster_id")
new_image = new_image.groupby(level=0).sum()
new_image.head()
3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 jboss-amq-6/amq63-openshift openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-grafana openshift/ose-jenkins openshift/ose-must-gather openshift/ose-oauth-proxy openshift/ose-tests openshift/ose-tools rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/redis-5 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/redis-32-rhel7 ubi8/dotnet-50 ubi8/ruby-27
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 0 0 0 0 0 0 5 6 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
00351e6e-53ce-465e-9493-cf0cd2367049 0 0 0 0 0 0 5 6 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
003ba133-e754-4d5a-bc57-675b386d1f05 0 0 0 0 0 0 5 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
00479ead-b7fc-49c2-ae20-3990a9b3d08c 0 0 0 0 0 0 5 6 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
00748c32-15c3-4586-98fb-e5078ca9f3b8 0 0 0 0 0 0 5 6 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
X_image = new_image
X_image.shape
(1810, 25)

The 25 unique categories for name and 1810 categories for cluster_id.

Encoding for name and cluster_id in container dataset

df_cont_name = df_cont[["cluster_id", "name"]]
dummy = pd.get_dummies(df_cont_name.name)
new_cont = pd.concat([df_cont_name, dummy], axis=1)
new_cont = new_cont.drop(columns="name")
new_cont = new_cont.set_index("cluster_id")
new_cont = new_cont.groupby(level=0).sum()
X_cont = new_cont
X_cont.head()
3scale-amp2/3scale-rhel7-operator 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 Calico Operator Cilium Cloud Native PostgreSQL Operator Elastic Cloud on Kubernetes F5 BIG-IP Controller Operator NGINX Ingress Operator NVIDIA GPU Operator Seldon Operator alertmanager amq7/amq-broker-rhel7-operator amq7/amq-streams-kafka-25-rhel7 amq7/amq-streams-kafka-26-rhel7 amq7/amq-streams-rhel7-operator amq7/amq-streams-rhel7-operator-metadata ansible-automation-platform/platform-resource-rhel7-operator ansible-tower cephcsi cert-manager cainjector cert-manager controller cert-manager webhook codeready-workspaces/devfileregistry-rhel8 codeready-workspaces/operator codeready-workspaces/pluginregistry-rhel8 codeready-workspaces/server-rhel8 collectd-exporter container-native-virtualization/bridge-marker container-native-virtualization/cluster-network-addons-operator container-native-virtualization/cnv-containernetworking-plugins container-native-virtualization/hostpath-provisioner-rhel8-operator container-native-virtualization/hyperconverged-cluster-operator container-native-virtualization/hyperconverged-cluster-webhook-rhel8 container-native-virtualization/kubemacpool container-native-virtualization/kubernetes-nmstate-handler-rhel8 container-native-virtualization/kubevirt-cpu-model-nfd-plugin container-native-virtualization/kubevirt-cpu-node-labeller container-native-virtualization/kubevirt-kvm-info-nfd-plugin container-native-virtualization/kubevirt-ssp-operator container-native-virtualization/node-maintenance-operator container-native-virtualization/ovs-cni-marker container-native-virtualization/ovs-cni-plugin container-native-virtualization/virt-api container-native-virtualization/virt-cdi-apiserver container-native-virtualization/virt-cdi-controller container-native-virtualization/virt-cdi-importer container-native-virtualization/virt-cdi-operator container-native-virtualization/virt-cdi-uploadproxy container-native-virtualization/virt-controller container-native-virtualization/virt-handler container-native-virtualization/virt-launcher container-native-virtualization/virt-operator container-native-virtualization/vm-import-controller-rhel8 container-native-virtualization/vm-import-operator-rhel8 costmanagement-metrics-operator distributed-tracing/jaeger-rhel8-operator grafana ibm common service webhook ibm-events-operator ibm-postgresql jboss-amq-6/amq63-openshift jboss-eap-7/eap73-rhel8-operator kube-state-metrics mcg-core mcg-operator mcr.microsoft.com/mssql/rhel8/server must-gather-service ocp-tools-4/odo-init-image ocs-operator ocs-registry openshift-compliance-content openshift-gitops-1-tech-preview/argocd-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8-operator openshift-gitops-1-tech-preview/kam-delivery-rhel8 openshift-gitops-1/gitops-rhel8 openshift-gitops-1/kam-delivery-rhel8 openshift-logging/cluster-logging-operator-bundle openshift-logging/cluster-logging-rhel8-operator openshift-logging/elasticsearch-operator-bundle openshift-logging/elasticsearch-proxy-rhel8 openshift-logging/elasticsearch-rhel8-operator openshift-logging/elasticsearch6-rhel8 openshift-logging/fluentd-rhel8 openshift-logging/kibana6-rhel8 openshift-pipelines-tech-preview/pipelines-controller-rhel8 openshift-pipelines-tech-preview/pipelines-operator-proxy-rhel8 openshift-pipelines-tech-preview/pipelines-rhel8-operator openshift-pipelines-tech-preview/pipelines-triggers-controller-rhel8 openshift-pipelines-tech-preview/pipelines-triggers-webhook-rhel8 openshift-pipelines-tech-preview/pipelines-webhook-rhel8 openshift-pipelines/pipelines-controller-rhel8 openshift-pipelines/pipelines-operator-proxy-rhel8 openshift-pipelines/pipelines-rhel8-operator openshift-pipelines/pipelines-triggers-controller-rhel8 openshift-pipelines/pipelines-triggers-core-interceptors-rhel8 openshift-pipelines/pipelines-triggers-eventlistenersink-rhel8 openshift-pipelines/pipelines-triggers-webhook-rhel8 openshift-pipelines/pipelines-webhook-rhel8 openshift-sandboxed-containers-operator openshift-serverless-1-tech-preview/eventing-kafka-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-dispatcher-rhel8 openshift-serverless-1/eventing-mtbroker-ingress-rhel8 openshift-serverless-1/eventing-mtchannel-broker-rhel8 openshift-serverless-1/eventing-mtping-rhel8 openshift-serverless-1/eventing-sugar-controller-rhel8 openshift-serverless-1/eventing-webhook-rhel8 openshift-serverless-1/ingress-rhel8-operator openshift-serverless-1/knative-rhel8-operator openshift-serverless-1/kourier-control-rhel8 openshift-serverless-1/serverless-rhel8-operator openshift-serverless-1/serving-activator-rhel8 openshift-serverless-1/serving-autoscaler-hpa-rhel8 openshift-serverless-1/serving-autoscaler-rhel8 openshift-serverless-1/serving-controller-rhel8 openshift-serverless-1/serving-domain-mapping-rhel8 openshift-serverless-1/serving-domain-mapping-webhook-rhel8 openshift-serverless-1/serving-queue-rhel8 openshift-service-mesh/galley-rhel8 openshift-service-mesh/grafana-rhel8 openshift-service-mesh/istio-cni-rhel8 openshift-service-mesh/istio-rhel8-operator openshift-service-mesh/kiali-rhel7 openshift-service-mesh/kiali-rhel8 openshift-service-mesh/kiali-rhel8-operator openshift-service-mesh/pilot-rhel8 openshift-service-mesh/prometheus-rhel8 openshift-service-mesh/proxyv2-rhel8 openshift/compliance-operator openshift/ose-aws-ebs-csi-driver openshift/ose-aws-ebs-csi-driver-operator openshift/ose-aws-machine-controllers openshift/ose-aws-pod-identity-webhook openshift/ose-azure-disk-csi-driver openshift/ose-azure-disk-csi-driver-operator openshift/ose-azure-machine-controllers openshift/ose-baremetal-machine-controllers openshift/ose-baremetal-operator openshift/ose-baremetal-runtimecfg openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-cloud-credential-operator openshift/ose-cluster-authentication-operator openshift/ose-cluster-autoscaler openshift/ose-cluster-autoscaler-operator openshift/ose-cluster-baremetal-operator openshift/ose-cluster-config-operator openshift/ose-cluster-csi-snapshot-controller-operator openshift/ose-cluster-dns-operator openshift/ose-cluster-etcd-operator openshift/ose-cluster-image-registry-operator openshift/ose-cluster-ingress-operator openshift/ose-cluster-kube-apiserver-operator openshift/ose-cluster-kube-controller-manager-operator openshift/ose-cluster-kube-descheduler-operator openshift/ose-cluster-kube-scheduler-operator openshift/ose-cluster-kube-storage-version-migrator-operator openshift/ose-cluster-logging-operator openshift/ose-cluster-machine-approver openshift/ose-cluster-monitoring-operator openshift/ose-cluster-network-operator openshift/ose-cluster-nfd-operator openshift/ose-cluster-node-tuning-operator openshift/ose-cluster-openshift-apiserver-operator openshift/ose-cluster-openshift-controller-manager-operator openshift/ose-cluster-policy-controller openshift/ose-cluster-samples-operator openshift/ose-cluster-storage-operator openshift/ose-clusterresourceoverride-rhel8-operator openshift/ose-configmap-reloader openshift/ose-console openshift/ose-console-operator openshift/ose-container-networking-plugins openshift/ose-coredns openshift/ose-csi-driver-manila openshift/ose-csi-driver-manila-operator openshift/ose-csi-driver-nfs openshift/ose-csi-external-attacher openshift/ose-csi-external-provisioner openshift/ose-csi-external-resizer openshift/ose-csi-external-snapshotter openshift/ose-csi-livenessprobe openshift/ose-csi-node-driver-registrar openshift/ose-csi-snapshot-controller openshift/ose-csi-snapshot-validation-webhook openshift/ose-deployer openshift/ose-docker-builder openshift/ose-docker-registry openshift/ose-egress-router-cni openshift/ose-etcd openshift/ose-gcp-machine-controllers openshift/ose-gcp-pd-csi-driver openshift/ose-gcp-pd-csi-driver-operator openshift/ose-grafana openshift/ose-haproxy-router openshift/ose-hyperkube openshift/ose-insights-operator openshift/ose-installer openshift/ose-ironic openshift/ose-ironic-inspector openshift/ose-ironic-ipa-downloader openshift/ose-ironic-machine-os-downloader openshift/ose-ironic-static-ip-manager openshift/ose-jenkins openshift/ose-k8s-prometheus-adapter openshift/ose-keepalived-ipfailover openshift/ose-kube-rbac-proxy openshift/ose-kube-state-metrics openshift/ose-kube-storage-version-migrator openshift/ose-kuryr-cni openshift/ose-kuryr-controller openshift/ose-libvirt-machine-controllers openshift/ose-local-storage-diskmaker openshift/ose-local-storage-operator openshift/ose-local-storage-operator-bundle openshift/ose-local-storage-static-provisioner openshift/ose-logging-elasticsearch6 openshift/ose-machine-api-operator openshift/ose-machine-config-operator openshift/ose-mdns-publisher openshift/ose-multus-admission-controller openshift/ose-multus-cni openshift/ose-multus-route-override-cni openshift/ose-multus-whereabouts-ipam-cni openshift/ose-must-gather openshift/ose-network-metrics-daemon openshift/ose-oauth-apiserver openshift/ose-oauth-proxy openshift/ose-oauth-server openshift/ose-openshift-apiserver openshift/ose-openshift-controller-manager openshift/ose-openshift-state-metrics openshift/ose-openstack-cinder-csi-driver openshift/ose-openstack-cinder-csi-driver-operator openshift/ose-openstack-machine-controllers openshift/ose-operator-lifecycle-manager openshift/ose-operator-marketplace openshift/ose-operator-registry openshift/ose-ovirt-csi-driver openshift/ose-ovirt-csi-driver-operator openshift/ose-ovirt-machine-controllers openshift/ose-ovn-kubernetes openshift/ose-prom-label-proxy openshift/ose-prometheus openshift/ose-prometheus-alertmanager openshift/ose-prometheus-config-reloader openshift/ose-prometheus-node-exporter openshift/ose-prometheus-operator openshift/ose-ptp openshift/ose-ptp-operator openshift/ose-sdn openshift/ose-service-ca-operator openshift/ose-sriov-cni openshift/ose-sriov-dp-admission-controller openshift/ose-sriov-infiniband-cni openshift/ose-sriov-network-config-daemon openshift/ose-sriov-network-device-plugin openshift/ose-sriov-network-operator openshift/ose-sriov-network-webhook openshift/ose-telemeter openshift/ose-template-service-broker openshift/ose-template-service-broker-operator openshift/ose-tests openshift/ose-thanos openshift/ose-tools openshift/ose-vertical-pod-autoscaler-rhel8-operator openshift/ose-vsphere-csi-driver openshift/ose-vsphere-csi-driver-operator openshift/ose-vsphere-csi-driver-syncer openshift/ose-vsphere-problem-detector openshift4/performance-addon-rhel8-operator quay/quay-container-security-operator-container quay/quay-container-security-operator-rhel8 quay/quay-operator-rhel8 rh-sso-7/sso74-openj9-openshift-rhel8 rhacm2/application-ui-rhel8 rhacm2/cert-manager-controller-rhel8 rhacm2/cert-policy-controller-rhel8 rhacm2/clusterlifecycle-state-metrics-rhel8 rhacm2/config-policy-controller-rhel8 rhacm2/console-api-rhel8 rhacm2/console-header-rhel8 rhacm2/console-rhel8 rhacm2/discovery-rhel8-operator rhacm2/endpoint-component-rhel8-operator rhacm2/endpoint-rhel8-operator rhacm2/governance-policy-propagator-rhel8 rhacm2/governance-policy-spec-sync-rhel8 rhacm2/governance-policy-status-sync-rhel8 rhacm2/governance-policy-template-sync-rhel8 rhacm2/grafana rhacm2/grc-ui-api-rhel8 rhacm2/grc-ui-rhel8 rhacm2/iam-policy-controller-rhel8 rhacm2/insights-client-rhel8 rhacm2/klusterlet-addon-controller-rhel8 rhacm2/klusterlet-addon-lease-controller-rhel8 rhacm2/klusterlet-addon-operator-rhel8 rhacm2/managedcluster-import-controller-rhel8 rhacm2/multicloud-manager-rhel8 rhacm2/multicluster-observability-rhel8-operator rhacm2/multicluster-operators-application-rhel8 rhacm2/multicluster-operators-channel-rhel8 rhacm2/multicluster-operators-deployable-rhel8 rhacm2/multicluster-operators-placementrule-rhel8 rhacm2/multicluster-operators-subscription-rhel8 rhacm2/multiclusterhub-repo-rhel8 rhacm2/multiclusterhub-rhel8 rhacm2/openshift-hive-rhel7 rhacm2/prometheus-alertmanager-rhel8 rhacm2/provider-credential-controller-rhel8 rhacm2/rcm-controller-rhel8 rhacm2/redisgraph-tls-rhel8 rhacm2/registration-rhel8 rhacm2/registration-rhel8-operator rhacm2/search-aggregator-rhel8 rhacm2/search-collector-rhel8 rhacm2/search-rhel8 rhacm2/search-ui-rhel8 rhacm2/submariner-addon-rhel8 rhacm2/thanos-receive-controller-rhel8 rhacm2/thanos-rhel7 rhacm2/work-rhel8 rhceph rhel7/couchbase-operator-admission rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/postgresql-96 rhel8/redis-5 rhmtc/openshift-migration-controller rhmtc/openshift-migration-operator rhmtc/openshift-migration-velero rhmtc/openshift-migration-velero-plugin-for-aws rhmtc/openshift-migration-velero-plugin-for-gcp rhmtc/openshift-migration-velero-plugin-for-microsoft-azure rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/postgresql-96-rhel7 rhscl/redis-32-rhel7 rook-ceph ubi8 ubi8/dotnet-50 ubi8/ruby-27 ubi8/ubi8-init volume-replication-operator
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 1 1 2 6 3 0 3 1 0 1 1 3 0 2 1 1 1 2 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 5 0 0 0 1 1 5 1 0 0 0 0 0 0 0 1 0 24 1 1 0 0 0 0 0 0 0 0 1 4 0 1 2 1 2 0 1 2 5 1 2 1 1 0 0 0 3 1 0 0 0 0 0 3 1 1 3 2 1 0 0 2 2 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
00067111-ddb4-437a-93e3-3d1c8ac25d31 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 5 2 0 1 1 0 1 1 1 1 1 1 1 1 6 3 0 1 1 0 1 1 2 0 2 1 1 1 0 1 0 0 0 0 1 2 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 5 0 0 0 0 1 3 1 0 7 2 2 2 2 0 0 1 18 0 1 0 0 0 0 0 0 0 0 4 2 0 1 2 1 2 0 1 2 1 0 2 1 0 0 0 0 3 1 0 0 0 0 8 0 0 0 0 2 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
002663ad-bcf4-4c7c-9530-ecb351fe4001 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
00351e6e-53ce-465e-9493-cf0cd2367049 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 1 1 2 6 3 0 3 1 0 1 1 3 0 2 1 1 1 2 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 5 0 0 0 1 1 5 1 0 0 0 0 0 0 0 1 0 24 1 1 0 0 0 0 0 0 0 0 1 4 0 1 2 1 2 0 1 2 5 1 2 1 1 0 0 0 3 1 0 0 0 0 0 3 1 1 3 2 1 0 0 2 2 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
003ba133-e754-4d5a-bc57-675b386d1f05 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 1 1 1 1 1 1 2 6 3 0 3 1 0 1 1 3 0 2 1 1 1 2 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 5 0 0 0 1 1 5 1 0 0 0 0 0 0 0 1 0 24 1 1 0 0 0 0 0 0 0 0 1 4 0 1 2 1 2 0 1 2 5 1 2 1 1 0 0 0 3 1 0 0 0 0 0 3 1 1 3 2 1 0 0 2 2 1 0 1 0 0 1 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 4 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
X_cont.shape
(1881, 353)

Here, the 353 unique name and 1881 cluster_id.

In the next step, we will perform the PCA in order to visualize different components in 2-D and 3-D.

3D plot for image_layer_dataset

pca3 = PCA(n_components=3)
pc3 = pca3.fit_transform(X_image)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pc3[:, 0],
    ys=pc3[:, 1],
    zs=pc3[:, 2],
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.title("PCA for image_layers dataset (3D)")
plt.show()
../_images/ML_clustering_52_0.png

Interactive 3D plot for image layer dataset

fig = px.scatter_3d(pc3, x=pc3[:, 0], y=pc3[:, 1], z=pc3[:, 2], opacity=0.5)
fig.show()

3D principal component analysis for container dataset

pcc3 = pca3.fit_transform(X_cont)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pcc3[:, 0],
    ys=pcc3[:, 1],
    zs=pcc3[:, 2],
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.title("PCA for containers dataset (3D)")
plt.show()
../_images/ML_clustering_57_0.png

3D interactive visualization

fig = px.scatter_3d(pcc3, x=pcc3[:, 0], y=pcc3[:, 1], z=pcc3[:, 2], opacity=1)
fig.show()

Now that we have visualized both image_layer dataset and container dataset in both 2D and 3D. We now implement Kmeans to identify different clusters in the set.

Before applying KMeans by inserting the number of clusters, we first find the optimal number of clusters in the dataset by using the Elbow method.

The Elbow Method

The elbow method is used to determine the optimal numbers of clusters in K-means clustering. The elbow method plots the value of inertia produced by different values of K.

Inertia is the within-cluster sum of squares criterion. It is a metric that shows how internally coherant the clusters are. It measures how well a dataset was clustered by K-Means. It is calculated by measuring the distance between each data point and its centroid, squaring this distance, and summing these squares across one cluster. A good model is one with low inertia AND a low number of clusters ( K ).

Applying the elbow method for the image layer dataset, we set the range of cluster from 1 - 10.

# How to find the best number of clusters?

no_of_clusters = range(1, 10)
inertia = []

for f in tqdm(no_of_clusters):
    kmeans = KMeans(n_clusters=f, random_state=2)
    kmeans = kmeans.fit(X_image)
    u = kmeans.inertia_
    inertia.append(u)
100%|██████████| 9/9 [00:07<00:00,  1.24it/s]

Inertia vs Number of cluster plot

fig, (ax1) = plt.subplots(1, figsize=(16, 6))
xx = np.arange(len(no_of_clusters))
ax1.plot(xx, inertia)
ax1.set_xticks(xx)
ax1.set_xticklabels(no_of_clusters, rotation="horizontal")
plt.xlabel("Number of clusters")
plt.ylabel("SSE")
plt.title("Inertia plot per K")
plt.show()
../_images/ML_clustering_66_0.png

From the above plot, it is seen that the optimal number of clusters for image layers dataset is 5. We will later apply it when implementing KMeans algorithm.

Next, we search for optimal number of clusters for container dataset.

# How to find the best number of clusters?

no_of_clusters = range(1, 10)
inertia = []

for f in tqdm(no_of_clusters):
    kmeans = KMeans(n_clusters=f, random_state=2)
    kmeans = kmeans.fit(X_cont)
    u = kmeans.inertia_
    inertia.append(u)
100%|██████████| 9/9 [00:17<00:00,  1.97s/it]
fig, (ax1) = plt.subplots(1, figsize=(16, 6))
xx = np.arange(len(no_of_clusters))
ax1.plot(xx, inertia)
ax1.set_xticks(xx)
ax1.set_xticklabels(no_of_clusters, rotation="horizontal")
plt.xlabel("Number of clusters")
plt.ylabel("SSE")
plt.title("Inertia plot per K")
plt.show()
../_images/ML_clustering_70_0.png

Based on the elbow plot, we will take 4 as the optimal number of clusters, which we will later apply through KMeans.

Implementing KMeans for image_layer_dataset

We take the number of clusters = 5 and apply the KMeans algorithm.

kmeans = KMeans(n_clusters=5, random_state=2)
kmeans = kmeans.fit(X_image)
clusters = kmeans.predict(X_image)
clusters
array([0, 0, 2, ..., 0, 0, 0], dtype=int32)
cluster_image_df = X_image
cluster_image_df["clusters"] = clusters
cluster_image_df.head()
3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 jboss-amq-6/amq63-openshift openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-grafana openshift/ose-jenkins openshift/ose-must-gather openshift/ose-oauth-proxy openshift/ose-tests openshift/ose-tools rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/redis-5 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/redis-32-rhel7 ubi8/dotnet-50 ubi8/ruby-27 clusters
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 0 0 0 0 0 0 5 6 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0
00351e6e-53ce-465e-9493-cf0cd2367049 0 0 0 0 0 0 5 6 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0
003ba133-e754-4d5a-bc57-675b386d1f05 0 0 0 0 0 0 5 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 2
00479ead-b7fc-49c2-ae20-3990a9b3d08c 0 0 0 0 0 0 5 6 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0
00748c32-15c3-4586-98fb-e5078ca9f3b8 0 0 0 0 0 0 5 6 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Visualizing the results of the 3D PCA

pca_i = pca3.fit_transform(cluster_image_df)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pca_i[:, 0], ys=pca_i[:, 1], zs=pca_i[:, 2], c=cluster_image_df["clusters"]
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.show()
../_images/ML_clustering_80_0.png

3D interactive visualization for image dataset

fig = px.scatter_3d(
    pca_i,
    x=pca_i[:, 0],
    y=pca_i[:, 1],
    z=pca_i[:, 2],
    color=cluster_image_df["clusters"],
    opacity=0.1,
)
fig.show()

Listing out the groups of image_name in different clusters

df1 = pd.DataFrame([])
global_mean = cluster_image_df.mean().sort_values(ascending=False)


for c in tqdm(range(0, 5)):
    c = cluster_image_df[cluster_image_df.clusters == c].iloc[:, :-1].mean()
    diff = c - global_mean
    d = diff.sort_values(ascending=False).head(20)
    d1 = pd.DataFrame(d).reset_index()
    d1.rename(columns={"index": "image_name"}, inplace=True)
    df1 = df1.append(d1.image_name)
df1 = df1.reset_index()
df1.rename(
    index={
        0: "cluster 1",
        1: "cluster 2",
        2: "cluster 3",
        3: "cluster 4",
        4: "cluster 5",
    },
    inplace=True,
)
df1.drop(columns="index", inplace=True)

df1
100%|██████████| 5/5 [00:00<00:00, 188.80it/s]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
cluster 1 openshift/ose-cli-artifacts openshift/ose-oauth-proxy openshift/ose-cli openshift/ose-jenkins rhscl/postgresql-10-rhel7 openshift/ose-tests 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 jboss-amq-6/amq63-openshift 3scale-amp2/backend-rhel7 openshift/ose-grafana rhel8/postgresql-10 ubi8/dotnet-50 rhscl/redis-32-rhel7 rhscl/mysql-57-rhel7 rhel8/redis-5 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/memcached-rhel7 rhscl/mongodb-36-rhel7 rhel8/httpd-24
cluster 2 openshift/ose-tools openshift/ose-must-gather rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-12 openshift/ose-cli-artifacts openshift/ose-oauth-proxy openshift/ose-cli 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/memcached-rhel7 ubi8/dotnet-50 rhscl/mysql-57-rhel7 rhel8/postgresql-10 rhel8/redis-5 rhscl/redis-32-rhel7 openshift/ose-grafana 3scale-amp2/backend-rhel7 jboss-amq-6/amq63-openshift 3scale-amp2/zync-rhel7 3scale-amp2/system-rhel7
cluster 3 ubi8/ruby-27 rhscl/mongodb-36-rhel7 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/memcached-rhel7 ubi8/dotnet-50 rhscl/redis-32-rhel7 rhscl/mysql-57-rhel7 rhel8/redis-5 rhel8/postgresql-10 openshift/ose-grafana 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 jboss-amq-6/amq63-openshift 3scale-amp2/backend-rhel7 openshift/ose-tests rhscl/postgresql-10-rhel7 rhel8/httpd-24 rhel8/mysql-80 openshift/ose-jenkins openshift/ose-tools
cluster 4 rhel8/postgresql-12 rhel8/mysql-80 openshift/ose-cli-artifacts openshift/ose-oauth-proxy openshift/ose-cli 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/memcached-rhel7 rhscl/redis-32-rhel7 rhel8/redis-5 rhscl/mysql-57-rhel7 rhel8/postgresql-10 ubi8/dotnet-50 openshift/ose-grafana 3scale-amp2/backend-rhel7 jboss-amq-6/amq63-openshift 3scale-amp2/zync-rhel7 3scale-amp2/system-rhel7 openshift/ose-tests rhel8/httpd-24 rhscl/postgresql-10-rhel7
cluster 5 openshift/ose-must-gather rhel8/postgresql-12 openshift/ose-cli-artifacts openshift/ose-oauth-proxy openshift/ose-cli 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/memcached-rhel7 rhel8/redis-5 rhscl/mysql-57-rhel7 rhscl/redis-32-rhel7 rhel8/postgresql-10 ubi8/dotnet-50 openshift/ose-grafana 3scale-amp2/backend-rhel7 jboss-amq-6/amq63-openshift 3scale-amp2/zync-rhel7 3scale-amp2/system-rhel7 openshift/ose-tests rhel8/httpd-24 rhscl/postgresql-10-rhel7

We generated the above dataframe by differencitating the global mean with the mean of the individual cluster, represented as a index in the dataframe. Each row in the dataframe df1, represents groups of top 20 image_names having a higher probability of occuring/running for the same user.

Implementing KMeans for container dataset

We take the number of clusters = 4. Because of the large number of unique features, the program will be computationally intensive. We apply KMeans algorithm by selecting 10 clusters.

kmeans = KMeans(n_clusters=4, random_state=2)
kmeans = kmeans.fit(X_cont)
clusters = kmeans.predict(X_cont)
clusters
array([0, 0, 3, ..., 0, 0, 1], dtype=int32)
cluster_cont_df = X_cont
cluster_cont_df["clusters"] = clusters
cluster_cont_df.head()
3scale-amp2/3scale-rhel7-operator 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 Calico Operator Cilium Cloud Native PostgreSQL Operator Elastic Cloud on Kubernetes F5 BIG-IP Controller Operator NGINX Ingress Operator NVIDIA GPU Operator Seldon Operator alertmanager amq7/amq-broker-rhel7-operator amq7/amq-streams-kafka-25-rhel7 amq7/amq-streams-kafka-26-rhel7 amq7/amq-streams-rhel7-operator amq7/amq-streams-rhel7-operator-metadata ansible-automation-platform/platform-resource-rhel7-operator ansible-tower cephcsi cert-manager cainjector cert-manager controller cert-manager webhook codeready-workspaces/devfileregistry-rhel8 codeready-workspaces/operator codeready-workspaces/pluginregistry-rhel8 codeready-workspaces/server-rhel8 collectd-exporter container-native-virtualization/bridge-marker container-native-virtualization/cluster-network-addons-operator container-native-virtualization/cnv-containernetworking-plugins container-native-virtualization/hostpath-provisioner-rhel8-operator container-native-virtualization/hyperconverged-cluster-operator container-native-virtualization/hyperconverged-cluster-webhook-rhel8 container-native-virtualization/kubemacpool container-native-virtualization/kubernetes-nmstate-handler-rhel8 container-native-virtualization/kubevirt-cpu-model-nfd-plugin container-native-virtualization/kubevirt-cpu-node-labeller container-native-virtualization/kubevirt-kvm-info-nfd-plugin container-native-virtualization/kubevirt-ssp-operator container-native-virtualization/node-maintenance-operator container-native-virtualization/ovs-cni-marker container-native-virtualization/ovs-cni-plugin container-native-virtualization/virt-api container-native-virtualization/virt-cdi-apiserver container-native-virtualization/virt-cdi-controller container-native-virtualization/virt-cdi-importer container-native-virtualization/virt-cdi-operator container-native-virtualization/virt-cdi-uploadproxy container-native-virtualization/virt-controller container-native-virtualization/virt-handler container-native-virtualization/virt-launcher container-native-virtualization/virt-operator container-native-virtualization/vm-import-controller-rhel8 container-native-virtualization/vm-import-operator-rhel8 costmanagement-metrics-operator distributed-tracing/jaeger-rhel8-operator grafana ibm common service webhook ibm-events-operator ibm-postgresql jboss-amq-6/amq63-openshift jboss-eap-7/eap73-rhel8-operator kube-state-metrics mcg-core mcg-operator mcr.microsoft.com/mssql/rhel8/server must-gather-service ocp-tools-4/odo-init-image ocs-operator ocs-registry openshift-compliance-content openshift-gitops-1-tech-preview/argocd-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8-operator openshift-gitops-1-tech-preview/kam-delivery-rhel8 openshift-gitops-1/gitops-rhel8 openshift-gitops-1/kam-delivery-rhel8 openshift-logging/cluster-logging-operator-bundle openshift-logging/cluster-logging-rhel8-operator openshift-logging/elasticsearch-operator-bundle openshift-logging/elasticsearch-proxy-rhel8 openshift-logging/elasticsearch-rhel8-operator openshift-logging/elasticsearch6-rhel8 openshift-logging/fluentd-rhel8 openshift-logging/kibana6-rhel8 openshift-pipelines-tech-preview/pipelines-controller-rhel8 openshift-pipelines-tech-preview/pipelines-operator-proxy-rhel8 openshift-pipelines-tech-preview/pipelines-rhel8-operator openshift-pipelines-tech-preview/pipelines-triggers-controller-rhel8 openshift-pipelines-tech-preview/pipelines-triggers-webhook-rhel8 openshift-pipelines-tech-preview/pipelines-webhook-rhel8 openshift-pipelines/pipelines-controller-rhel8 openshift-pipelines/pipelines-operator-proxy-rhel8 openshift-pipelines/pipelines-rhel8-operator openshift-pipelines/pipelines-triggers-controller-rhel8 openshift-pipelines/pipelines-triggers-core-interceptors-rhel8 openshift-pipelines/pipelines-triggers-eventlistenersink-rhel8 openshift-pipelines/pipelines-triggers-webhook-rhel8 openshift-pipelines/pipelines-webhook-rhel8 openshift-sandboxed-containers-operator openshift-serverless-1-tech-preview/eventing-kafka-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-dispatcher-rhel8 openshift-serverless-1/eventing-mtbroker-ingress-rhel8 openshift-serverless-1/eventing-mtchannel-broker-rhel8 openshift-serverless-1/eventing-mtping-rhel8 openshift-serverless-1/eventing-sugar-controller-rhel8 openshift-serverless-1/eventing-webhook-rhel8 openshift-serverless-1/ingress-rhel8-operator openshift-serverless-1/knative-rhel8-operator openshift-serverless-1/kourier-control-rhel8 openshift-serverless-1/serverless-rhel8-operator openshift-serverless-1/serving-activator-rhel8 openshift-serverless-1/serving-autoscaler-hpa-rhel8 openshift-serverless-1/serving-autoscaler-rhel8 openshift-serverless-1/serving-controller-rhel8 openshift-serverless-1/serving-domain-mapping-rhel8 openshift-serverless-1/serving-domain-mapping-webhook-rhel8 openshift-serverless-1/serving-queue-rhel8 openshift-service-mesh/galley-rhel8 openshift-service-mesh/grafana-rhel8 openshift-service-mesh/istio-cni-rhel8 openshift-service-mesh/istio-rhel8-operator openshift-service-mesh/kiali-rhel7 openshift-service-mesh/kiali-rhel8 openshift-service-mesh/kiali-rhel8-operator openshift-service-mesh/pilot-rhel8 openshift-service-mesh/prometheus-rhel8 openshift-service-mesh/proxyv2-rhel8 openshift/compliance-operator openshift/ose-aws-ebs-csi-driver openshift/ose-aws-ebs-csi-driver-operator openshift/ose-aws-machine-controllers openshift/ose-aws-pod-identity-webhook openshift/ose-azure-disk-csi-driver openshift/ose-azure-disk-csi-driver-operator openshift/ose-azure-machine-controllers openshift/ose-baremetal-machine-controllers openshift/ose-baremetal-operator openshift/ose-baremetal-runtimecfg openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-cloud-credential-operator openshift/ose-cluster-authentication-operator openshift/ose-cluster-autoscaler openshift/ose-cluster-autoscaler-operator openshift/ose-cluster-baremetal-operator openshift/ose-cluster-config-operator openshift/ose-cluster-csi-snapshot-controller-operator openshift/ose-cluster-dns-operator openshift/ose-cluster-etcd-operator openshift/ose-cluster-image-registry-operator openshift/ose-cluster-ingress-operator openshift/ose-cluster-kube-apiserver-operator openshift/ose-cluster-kube-controller-manager-operator openshift/ose-cluster-kube-descheduler-operator openshift/ose-cluster-kube-scheduler-operator openshift/ose-cluster-kube-storage-version-migrator-operator openshift/ose-cluster-logging-operator openshift/ose-cluster-machine-approver openshift/ose-cluster-monitoring-operator openshift/ose-cluster-network-operator openshift/ose-cluster-nfd-operator openshift/ose-cluster-node-tuning-operator openshift/ose-cluster-openshift-apiserver-operator openshift/ose-cluster-openshift-controller-manager-operator openshift/ose-cluster-policy-controller openshift/ose-cluster-samples-operator openshift/ose-cluster-storage-operator openshift/ose-clusterresourceoverride-rhel8-operator openshift/ose-configmap-reloader openshift/ose-console openshift/ose-console-operator openshift/ose-container-networking-plugins openshift/ose-coredns openshift/ose-csi-driver-manila openshift/ose-csi-driver-manila-operator openshift/ose-csi-driver-nfs openshift/ose-csi-external-attacher openshift/ose-csi-external-provisioner openshift/ose-csi-external-resizer openshift/ose-csi-external-snapshotter openshift/ose-csi-livenessprobe openshift/ose-csi-node-driver-registrar openshift/ose-csi-snapshot-controller openshift/ose-csi-snapshot-validation-webhook openshift/ose-deployer openshift/ose-docker-builder openshift/ose-docker-registry openshift/ose-egress-router-cni openshift/ose-etcd openshift/ose-gcp-machine-controllers openshift/ose-gcp-pd-csi-driver openshift/ose-gcp-pd-csi-driver-operator openshift/ose-grafana openshift/ose-haproxy-router openshift/ose-hyperkube openshift/ose-insights-operator openshift/ose-installer openshift/ose-ironic openshift/ose-ironic-inspector openshift/ose-ironic-ipa-downloader openshift/ose-ironic-machine-os-downloader openshift/ose-ironic-static-ip-manager openshift/ose-jenkins openshift/ose-k8s-prometheus-adapter openshift/ose-keepalived-ipfailover openshift/ose-kube-rbac-proxy openshift/ose-kube-state-metrics openshift/ose-kube-storage-version-migrator openshift/ose-kuryr-cni openshift/ose-kuryr-controller openshift/ose-libvirt-machine-controllers openshift/ose-local-storage-diskmaker openshift/ose-local-storage-operator openshift/ose-local-storage-operator-bundle openshift/ose-local-storage-static-provisioner openshift/ose-logging-elasticsearch6 openshift/ose-machine-api-operator openshift/ose-machine-config-operator openshift/ose-mdns-publisher openshift/ose-multus-admission-controller openshift/ose-multus-cni openshift/ose-multus-route-override-cni openshift/ose-multus-whereabouts-ipam-cni openshift/ose-must-gather openshift/ose-network-metrics-daemon openshift/ose-oauth-apiserver openshift/ose-oauth-proxy openshift/ose-oauth-server openshift/ose-openshift-apiserver openshift/ose-openshift-controller-manager openshift/ose-openshift-state-metrics openshift/ose-openstack-cinder-csi-driver openshift/ose-openstack-cinder-csi-driver-operator openshift/ose-openstack-machine-controllers openshift/ose-operator-lifecycle-manager openshift/ose-operator-marketplace openshift/ose-operator-registry openshift/ose-ovirt-csi-driver openshift/ose-ovirt-csi-driver-operator openshift/ose-ovirt-machine-controllers openshift/ose-ovn-kubernetes openshift/ose-prom-label-proxy openshift/ose-prometheus openshift/ose-prometheus-alertmanager openshift/ose-prometheus-config-reloader openshift/ose-prometheus-node-exporter openshift/ose-prometheus-operator openshift/ose-ptp openshift/ose-ptp-operator openshift/ose-sdn openshift/ose-service-ca-operator openshift/ose-sriov-cni openshift/ose-sriov-dp-admission-controller openshift/ose-sriov-infiniband-cni openshift/ose-sriov-network-config-daemon openshift/ose-sriov-network-device-plugin openshift/ose-sriov-network-operator openshift/ose-sriov-network-webhook openshift/ose-telemeter openshift/ose-template-service-broker openshift/ose-template-service-broker-operator openshift/ose-tests openshift/ose-thanos openshift/ose-tools openshift/ose-vertical-pod-autoscaler-rhel8-operator openshift/ose-vsphere-csi-driver openshift/ose-vsphere-csi-driver-operator openshift/ose-vsphere-csi-driver-syncer openshift/ose-vsphere-problem-detector openshift4/performance-addon-rhel8-operator quay/quay-container-security-operator-container quay/quay-container-security-operator-rhel8 quay/quay-operator-rhel8 rh-sso-7/sso74-openj9-openshift-rhel8 rhacm2/application-ui-rhel8 rhacm2/cert-manager-controller-rhel8 rhacm2/cert-policy-controller-rhel8 rhacm2/clusterlifecycle-state-metrics-rhel8 rhacm2/config-policy-controller-rhel8 rhacm2/console-api-rhel8 rhacm2/console-header-rhel8 rhacm2/console-rhel8 rhacm2/discovery-rhel8-operator rhacm2/endpoint-component-rhel8-operator rhacm2/endpoint-rhel8-operator rhacm2/governance-policy-propagator-rhel8 rhacm2/governance-policy-spec-sync-rhel8 rhacm2/governance-policy-status-sync-rhel8 rhacm2/governance-policy-template-sync-rhel8 rhacm2/grafana rhacm2/grc-ui-api-rhel8 rhacm2/grc-ui-rhel8 rhacm2/iam-policy-controller-rhel8 rhacm2/insights-client-rhel8 rhacm2/klusterlet-addon-controller-rhel8 rhacm2/klusterlet-addon-lease-controller-rhel8 rhacm2/klusterlet-addon-operator-rhel8 rhacm2/managedcluster-import-controller-rhel8 rhacm2/multicloud-manager-rhel8 rhacm2/multicluster-observability-rhel8-operator rhacm2/multicluster-operators-application-rhel8 rhacm2/multicluster-operators-channel-rhel8 rhacm2/multicluster-operators-deployable-rhel8 rhacm2/multicluster-operators-placementrule-rhel8 rhacm2/multicluster-operators-subscription-rhel8 rhacm2/multiclusterhub-repo-rhel8 rhacm2/multiclusterhub-rhel8 rhacm2/openshift-hive-rhel7 rhacm2/prometheus-alertmanager-rhel8 rhacm2/provider-credential-controller-rhel8 rhacm2/rcm-controller-rhel8 rhacm2/redisgraph-tls-rhel8 rhacm2/registration-rhel8 rhacm2/registration-rhel8-operator rhacm2/search-aggregator-rhel8 rhacm2/search-collector-rhel8 rhacm2/search-rhel8 rhacm2/search-ui-rhel8 rhacm2/submariner-addon-rhel8 rhacm2/thanos-receive-controller-rhel8 rhacm2/thanos-rhel7 rhacm2/work-rhel8 rhceph rhel7/couchbase-operator-admission rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/postgresql-96 rhel8/redis-5 rhmtc/openshift-migration-controller rhmtc/openshift-migration-operator rhmtc/openshift-migration-velero rhmtc/openshift-migration-velero-plugin-for-aws rhmtc/openshift-migration-velero-plugin-for-gcp rhmtc/openshift-migration-velero-plugin-for-microsoft-azure rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/postgresql-96-rhel7 rhscl/redis-32-rhel7 rook-ceph ubi8 ubi8/dotnet-50 ubi8/ruby-27 ubi8/ubi8-init volume-replication-operator clusters
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 1 1 2 6 3 0 3 1 0 1 1 3 0 2 1 1 1 2 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 5 0 0 0 1 1 5 1 0 0 0 0 0 0 0 1 0 24 1 1 0 0 0 0 0 0 0 0 1 4 0 1 2 1 2 0 1 2 5 1 2 1 1 0 0 0 3 1 0 0 0 0 0 3 1 1 3 2 1 0 0 2 2 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
00067111-ddb4-437a-93e3-3d1c8ac25d31 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 5 2 0 1 1 0 1 1 1 1 1 1 1 1 6 3 0 1 1 0 1 1 2 0 2 1 1 1 0 1 0 0 0 0 1 2 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 5 0 0 0 0 1 3 1 0 7 2 2 2 2 0 0 1 18 0 1 0 0 0 0 0 0 0 0 4 2 0 1 2 1 2 0 1 2 1 0 2 1 0 0 0 0 3 1 0 0 0 0 8 0 0 0 0 2 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
002663ad-bcf4-4c7c-9530-ecb351fe4001 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3
00351e6e-53ce-465e-9493-cf0cd2367049 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 1 1 2 6 3 0 3 1 0 1 1 3 0 2 1 1 1 2 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 5 0 0 0 1 1 5 1 0 0 0 0 0 0 0 1 0 24 1 1 0 0 0 0 0 0 0 0 1 4 0 1 2 1 2 0 1 2 5 1 2 1 1 0 0 0 3 1 0 0 0 0 0 3 1 1 3 2 1 0 0 2 2 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
003ba133-e754-4d5a-bc57-675b386d1f05 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 1 1 1 1 1 1 2 6 3 0 3 1 0 1 1 3 0 2 1 1 1 2 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 5 0 0 0 1 1 5 1 0 0 0 0 0 0 0 1 0 24 1 1 0 0 0 0 0 0 0 0 1 4 0 1 2 1 2 0 1 2 5 1 2 1 1 0 0 0 3 1 0 0 0 0 0 3 1 1 3 2 1 0 0 2 2 1 0 1 0 0 1 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 4 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Visualizing the results of the 3D PCA

pca_c = pca3.fit_transform(cluster_cont_df)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pca_c[:, 0], ys=pca_c[:, 1], zs=pca_c[:, 2], c=cluster_cont_df["clusters"]
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.show()
../_images/ML_clustering_94_0.png

3D interactive visualization for image dataset

fig = px.scatter_3d(
    pca_c,
    x=pca_c[:, 0],
    y=pca_c[:, 1],
    z=pca_c[:, 2],
    color=cluster_cont_df["clusters"],
    opacity=0.1,
)
fig.show()

Listing out the groups of image_name in different clusters

df1 = pd.DataFrame([])
global_mean = cluster_cont_df.mean().sort_values(ascending=False)


for c in tqdm(range(0, 4)):
    c = cluster_cont_df[cluster_cont_df.clusters == c].iloc[:, :-1].mean()
    diff = c - global_mean
    d = diff.sort_values(ascending=False).head(20)
    d1 = pd.DataFrame(d).reset_index()
    d1.rename(columns={"index": "image_name"}, inplace=True)
    df1 = df1.append(d1.image_name)
df1 = df1.reset_index()
df1.rename(
    index={0: "cluster 1", 1: "cluster 2", 2: "cluster 3", 3: "cluster 4"}, inplace=True
)
df1.drop(columns="index", inplace=True)

df1
100%|██████████| 4/4 [00:00<00:00, 177.03it/s]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
cluster 1 rhacm2/work-rhel8 rhacm2/registration-rhel8 rhacm2/registration-rhel8-operator rhacm2/multicloud-manager-rhel8 openshift/ose-cluster-kube-apiserver-operator openshift/ose-hyperkube openshift/ose-etcd openshift/ose-thanos openshift/ose-sdn openshift/ose-machine-config-operator openshift/ose-operator-lifecycle-manager openshift/ose-cluster-kube-controller-manager-... openshift/ose-cluster-network-operator openshift/ose-cluster-kube-scheduler-operator openshift/ose-oauth-proxy openshift/ose-service-ca-operator openshift/ose-oauth-apiserver openshift/ose-prometheus-node-exporter openshift/ose-openshift-apiserver openshift/ose-cluster-node-tuning-operator
cluster 2 openshift/ose-kube-rbac-proxy openshift/ose-machine-api-operator openshift/ose-baremetal-runtimecfg openshift/ose-ironic openshift/ose-ovn-kubernetes openshift/ose-csi-livenessprobe openshift/ose-cli openshift/ose-aws-ebs-csi-driver openshift/ose-ironic-inspector openshift/ose-ironic-ipa-downloader openshift/ose-ironic-machine-os-downloader openshift/ose-csi-external-provisioner openshift/ose-csi-node-driver-registrar openshift/ose-csi-external-attacher openshift/ose-haproxy-router openshift/ose-csi-external-resizer openshift/ose-coredns openshift/ose-csi-external-snapshotter openshift/ose-keepalived-ipfailover openshift/ose-aws-pod-identity-webhook
cluster 3 rhceph openshift/ose-kube-rbac-proxy openshift/ose-baremetal-runtimecfg openshift/ose-csi-external-snapshotter openshift/ose-csi-node-driver-registrar openshift/ose-machine-api-operator openshift/ose-csi-external-provisioner openshift/ose-csi-driver-manila openshift/ose-csi-driver-nfs openshift/ose-openstack-cinder-csi-driver rook-ceph rhel8/postgresql-12 cephcsi openshift/ose-local-storage-diskmaker openshift/ose-local-storage-operator openshift/ose-mdns-publisher openshift/ose-cli openshift/ose-csi-external-resizer container-native-virtualization/virt-launcher openshift/ose-csi-external-attacher
cluster 4 openshift/ose-local-storage-operator openshift/ose-local-storage-diskmaker openshift/ose-local-storage-operator-bundle container-native-virtualization/kubevirt-cpu-n... container-native-virtualization/virt-operator container-native-virtualization/virt-controller container-native-virtualization/virt-handler container-native-virtualization/kubernetes-nms... container-native-virtualization/kubemacpool ubi8/dotnet-50 Cilium openshift-serverless-1/serving-queue-rhel8 container-native-virtualization/node-maintenan... container-native-virtualization/cnv-containern... container-native-virtualization/vm-import-cont... container-native-virtualization/virt-cdi-uploa... container-native-virtualization/kubevirt-cpu-m... container-native-virtualization/kubevirt-kvm-i... container-native-virtualization/virt-cdi-operator container-native-virtualization/virt-api

We generated the above dataframe by differencitating the global mean with the mean of the individual cluster, represented as a index in the dataframe. Each row in the dataframe df1, represents groups of top 20 image_names having a higher probability of occuring/running for the same user.



KMeans clustering by similarity encoders: (Taking frequency into considerations)

Clustering categorical features are challenging due to lack of labels required for analysing the data. Therefore, it becomes essential to apply the right encoding scheme for corresponding features. Pros and cons of different encode process are descried in this notebook.

For the feature, name resembling different product names is a nominal categorical feature. It is a challenge to balance between performance and explainability. In this analysis, we are following word based encoders, i.e., Similarity Encoders which extracts the information from the n-grams(collection of characters) present in the strings.

Encoding Image_layer dataset

In this issue, we are applying the encoding for the image_name from the dataset. Similar to previous approach, we encoded the image_name with respect to the cluster_id.

Similarity encoding

df_image_name = df_image[["name"]].reindex()
enc = SimilarityEncoder(similarity="ngram")
X_enc = enc.fit_transform(df_image_name)
X_image_name = pd.DataFrame(X_enc)
X_image_name.columns = sorted(df_image_name.name.unique())
X_image = pd.concat([df_image["cluster_id"], X_image_name], axis=1)
X_image = X_image.set_index("cluster_id")
X_image = X_image.groupby(level=0).sum()
X_image.head()
3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 jboss-amq-6/amq63-openshift openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-grafana openshift/ose-jenkins openshift/ose-must-gather openshift/ose-oauth-proxy openshift/ose-tests openshift/ose-tools rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/redis-5 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/redis-32-rhel7 ubi8/dotnet-50 ubi8/ruby-27
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 0.124094 0.292594 0.177182 0.147851 0.154433 2.684535 10.221675 10.190476 6.483450 6.483450 5.695925 8.241379 7.128506 6.952486 0.0 0.0 0.121059 0.121059 0.0 0.084199 0.088312 0.185413 0.088312 0.0 0.0
00351e6e-53ce-465e-9493-cf0cd2367049 0.124094 0.292594 0.177182 0.147851 0.154433 2.684535 10.221675 10.190476 6.483450 6.483450 5.695925 8.241379 7.128506 6.952486 0.0 0.0 0.121059 0.121059 0.0 0.084199 0.088312 0.185413 0.088312 0.0 0.0
003ba133-e754-4d5a-bc57-675b386d1f05 0.056678 0.134699 0.063974 0.068381 0.071675 1.691729 6.793103 4.190476 4.175758 4.175758 3.695925 6.241379 4.503506 4.503506 0.0 0.0 0.077263 0.077263 0.0 0.043103 0.045455 0.107491 0.045455 0.0 0.0
00479ead-b7fc-49c2-ae20-3990a9b3d08c 0.124094 0.292594 0.177182 0.147851 0.154433 2.684535 10.221675 10.190476 6.483450 6.483450 5.695925 8.241379 7.128506 6.952486 0.0 0.0 0.121059 0.121059 0.0 0.084199 0.088312 0.185413 0.088312 0.0 0.0
00748c32-15c3-4586-98fb-e5078ca9f3b8 0.124094 0.292594 0.177182 0.147851 0.154433 2.684535 10.221675 10.190476 6.483450 6.483450 5.695925 8.241379 7.128506 6.952486 0.0 0.0 0.121059 0.121059 0.0 0.084199 0.088312 0.185413 0.088312 0.0 0.0
X_image.shape
(1810, 25)

1810 rows corresponds to 1810 cluster_id.

Encoding container dataset

df_cont_name = df_cont[["name"]].reindex()
enc = SimilarityEncoder(similarity="ngram")
X_enc = enc.fit_transform(df_cont_name)
X_cont_name = pd.DataFrame(X_enc)
X_cont_name.columns = sorted(df_cont_name.name.unique())
X_cont = pd.concat([df_cont["cluster_id"], X_cont_name], axis=1)
X_cont = X_cont.set_index("cluster_id")
X_cont = X_cont.groupby(level=0).sum()
X_cont.head()
3scale-amp2/3scale-rhel7-operator 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 Calico Operator Cilium Cloud Native PostgreSQL Operator Elastic Cloud on Kubernetes F5 BIG-IP Controller Operator NGINX Ingress Operator NVIDIA GPU Operator Seldon Operator alertmanager amq7/amq-broker-rhel7-operator amq7/amq-streams-kafka-25-rhel7 amq7/amq-streams-kafka-26-rhel7 amq7/amq-streams-rhel7-operator amq7/amq-streams-rhel7-operator-metadata ansible-automation-platform/platform-resource-rhel7-operator ansible-tower cephcsi cert-manager cainjector cert-manager controller cert-manager webhook codeready-workspaces/devfileregistry-rhel8 codeready-workspaces/operator codeready-workspaces/pluginregistry-rhel8 codeready-workspaces/server-rhel8 collectd-exporter container-native-virtualization/bridge-marker container-native-virtualization/cluster-network-addons-operator container-native-virtualization/cnv-containernetworking-plugins container-native-virtualization/hostpath-provisioner-rhel8-operator container-native-virtualization/hyperconverged-cluster-operator container-native-virtualization/hyperconverged-cluster-webhook-rhel8 container-native-virtualization/kubemacpool container-native-virtualization/kubernetes-nmstate-handler-rhel8 container-native-virtualization/kubevirt-cpu-model-nfd-plugin container-native-virtualization/kubevirt-cpu-node-labeller container-native-virtualization/kubevirt-kvm-info-nfd-plugin container-native-virtualization/kubevirt-ssp-operator container-native-virtualization/node-maintenance-operator container-native-virtualization/ovs-cni-marker container-native-virtualization/ovs-cni-plugin container-native-virtualization/virt-api container-native-virtualization/virt-cdi-apiserver container-native-virtualization/virt-cdi-controller container-native-virtualization/virt-cdi-importer container-native-virtualization/virt-cdi-operator container-native-virtualization/virt-cdi-uploadproxy container-native-virtualization/virt-controller container-native-virtualization/virt-handler container-native-virtualization/virt-launcher container-native-virtualization/virt-operator container-native-virtualization/vm-import-controller-rhel8 container-native-virtualization/vm-import-operator-rhel8 costmanagement-metrics-operator distributed-tracing/jaeger-rhel8-operator grafana ibm common service webhook ibm-events-operator ibm-postgresql jboss-amq-6/amq63-openshift jboss-eap-7/eap73-rhel8-operator kube-state-metrics mcg-core mcg-operator mcr.microsoft.com/mssql/rhel8/server must-gather-service ocp-tools-4/odo-init-image ocs-operator ocs-registry openshift-compliance-content openshift-gitops-1-tech-preview/argocd-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8-operator openshift-gitops-1-tech-preview/kam-delivery-rhel8 openshift-gitops-1/gitops-rhel8 openshift-gitops-1/kam-delivery-rhel8 openshift-logging/cluster-logging-operator-bundle openshift-logging/cluster-logging-rhel8-operator openshift-logging/elasticsearch-operator-bundle openshift-logging/elasticsearch-proxy-rhel8 openshift-logging/elasticsearch-rhel8-operator openshift-logging/elasticsearch6-rhel8 openshift-logging/fluentd-rhel8 openshift-logging/kibana6-rhel8 openshift-pipelines-tech-preview/pipelines-controller-rhel8 openshift-pipelines-tech-preview/pipelines-operator-proxy-rhel8 openshift-pipelines-tech-preview/pipelines-rhel8-operator openshift-pipelines-tech-preview/pipelines-triggers-controller-rhel8 openshift-pipelines-tech-preview/pipelines-triggers-webhook-rhel8 openshift-pipelines-tech-preview/pipelines-webhook-rhel8 openshift-pipelines/pipelines-controller-rhel8 openshift-pipelines/pipelines-operator-proxy-rhel8 openshift-pipelines/pipelines-rhel8-operator openshift-pipelines/pipelines-triggers-controller-rhel8 openshift-pipelines/pipelines-triggers-core-interceptors-rhel8 openshift-pipelines/pipelines-triggers-eventlistenersink-rhel8 openshift-pipelines/pipelines-triggers-webhook-rhel8 openshift-pipelines/pipelines-webhook-rhel8 openshift-sandboxed-containers-operator openshift-serverless-1-tech-preview/eventing-kafka-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-dispatcher-rhel8 openshift-serverless-1/eventing-mtbroker-ingress-rhel8 openshift-serverless-1/eventing-mtchannel-broker-rhel8 openshift-serverless-1/eventing-mtping-rhel8 openshift-serverless-1/eventing-sugar-controller-rhel8 openshift-serverless-1/eventing-webhook-rhel8 openshift-serverless-1/ingress-rhel8-operator openshift-serverless-1/knative-rhel8-operator openshift-serverless-1/kourier-control-rhel8 openshift-serverless-1/serverless-rhel8-operator openshift-serverless-1/serving-activator-rhel8 openshift-serverless-1/serving-autoscaler-hpa-rhel8 openshift-serverless-1/serving-autoscaler-rhel8 openshift-serverless-1/serving-controller-rhel8 openshift-serverless-1/serving-domain-mapping-rhel8 openshift-serverless-1/serving-domain-mapping-webhook-rhel8 openshift-serverless-1/serving-queue-rhel8 openshift-service-mesh/galley-rhel8 openshift-service-mesh/grafana-rhel8 openshift-service-mesh/istio-cni-rhel8 openshift-service-mesh/istio-rhel8-operator openshift-service-mesh/kiali-rhel7 openshift-service-mesh/kiali-rhel8 openshift-service-mesh/kiali-rhel8-operator openshift-service-mesh/pilot-rhel8 openshift-service-mesh/prometheus-rhel8 openshift-service-mesh/proxyv2-rhel8 openshift/compliance-operator openshift/ose-aws-ebs-csi-driver openshift/ose-aws-ebs-csi-driver-operator openshift/ose-aws-machine-controllers openshift/ose-aws-pod-identity-webhook openshift/ose-azure-disk-csi-driver openshift/ose-azure-disk-csi-driver-operator openshift/ose-azure-machine-controllers openshift/ose-baremetal-machine-controllers openshift/ose-baremetal-operator openshift/ose-baremetal-runtimecfg openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-cloud-credential-operator openshift/ose-cluster-authentication-operator openshift/ose-cluster-autoscaler openshift/ose-cluster-autoscaler-operator openshift/ose-cluster-baremetal-operator openshift/ose-cluster-config-operator openshift/ose-cluster-csi-snapshot-controller-operator openshift/ose-cluster-dns-operator openshift/ose-cluster-etcd-operator openshift/ose-cluster-image-registry-operator openshift/ose-cluster-ingress-operator openshift/ose-cluster-kube-apiserver-operator openshift/ose-cluster-kube-controller-manager-operator openshift/ose-cluster-kube-descheduler-operator openshift/ose-cluster-kube-scheduler-operator openshift/ose-cluster-kube-storage-version-migrator-operator openshift/ose-cluster-logging-operator openshift/ose-cluster-machine-approver openshift/ose-cluster-monitoring-operator openshift/ose-cluster-network-operator openshift/ose-cluster-nfd-operator openshift/ose-cluster-node-tuning-operator openshift/ose-cluster-openshift-apiserver-operator openshift/ose-cluster-openshift-controller-manager-operator openshift/ose-cluster-policy-controller openshift/ose-cluster-samples-operator openshift/ose-cluster-storage-operator openshift/ose-clusterresourceoverride-rhel8-operator openshift/ose-configmap-reloader openshift/ose-console openshift/ose-console-operator openshift/ose-container-networking-plugins openshift/ose-coredns openshift/ose-csi-driver-manila openshift/ose-csi-driver-manila-operator openshift/ose-csi-driver-nfs openshift/ose-csi-external-attacher openshift/ose-csi-external-provisioner openshift/ose-csi-external-resizer openshift/ose-csi-external-snapshotter openshift/ose-csi-livenessprobe openshift/ose-csi-node-driver-registrar openshift/ose-csi-snapshot-controller openshift/ose-csi-snapshot-validation-webhook openshift/ose-deployer openshift/ose-docker-builder openshift/ose-docker-registry openshift/ose-egress-router-cni openshift/ose-etcd openshift/ose-gcp-machine-controllers openshift/ose-gcp-pd-csi-driver openshift/ose-gcp-pd-csi-driver-operator openshift/ose-grafana openshift/ose-haproxy-router openshift/ose-hyperkube openshift/ose-insights-operator openshift/ose-installer openshift/ose-ironic openshift/ose-ironic-inspector openshift/ose-ironic-ipa-downloader openshift/ose-ironic-machine-os-downloader openshift/ose-ironic-static-ip-manager openshift/ose-jenkins openshift/ose-k8s-prometheus-adapter openshift/ose-keepalived-ipfailover openshift/ose-kube-rbac-proxy openshift/ose-kube-state-metrics openshift/ose-kube-storage-version-migrator openshift/ose-kuryr-cni openshift/ose-kuryr-controller openshift/ose-libvirt-machine-controllers openshift/ose-local-storage-diskmaker openshift/ose-local-storage-operator openshift/ose-local-storage-operator-bundle openshift/ose-local-storage-static-provisioner openshift/ose-logging-elasticsearch6 openshift/ose-machine-api-operator openshift/ose-machine-config-operator openshift/ose-mdns-publisher openshift/ose-multus-admission-controller openshift/ose-multus-cni openshift/ose-multus-route-override-cni openshift/ose-multus-whereabouts-ipam-cni openshift/ose-must-gather openshift/ose-network-metrics-daemon openshift/ose-oauth-apiserver openshift/ose-oauth-proxy openshift/ose-oauth-server openshift/ose-openshift-apiserver openshift/ose-openshift-controller-manager openshift/ose-openshift-state-metrics openshift/ose-openstack-cinder-csi-driver openshift/ose-openstack-cinder-csi-driver-operator openshift/ose-openstack-machine-controllers openshift/ose-operator-lifecycle-manager openshift/ose-operator-marketplace openshift/ose-operator-registry openshift/ose-ovirt-csi-driver openshift/ose-ovirt-csi-driver-operator openshift/ose-ovirt-machine-controllers openshift/ose-ovn-kubernetes openshift/ose-prom-label-proxy openshift/ose-prometheus openshift/ose-prometheus-alertmanager openshift/ose-prometheus-config-reloader openshift/ose-prometheus-node-exporter openshift/ose-prometheus-operator openshift/ose-ptp openshift/ose-ptp-operator openshift/ose-sdn openshift/ose-service-ca-operator openshift/ose-sriov-cni openshift/ose-sriov-dp-admission-controller openshift/ose-sriov-infiniband-cni openshift/ose-sriov-network-config-daemon openshift/ose-sriov-network-device-plugin openshift/ose-sriov-network-operator openshift/ose-sriov-network-webhook openshift/ose-telemeter openshift/ose-template-service-broker openshift/ose-template-service-broker-operator openshift/ose-tests openshift/ose-thanos openshift/ose-tools openshift/ose-vertical-pod-autoscaler-rhel8-operator openshift/ose-vsphere-csi-driver openshift/ose-vsphere-csi-driver-operator openshift/ose-vsphere-csi-driver-syncer openshift/ose-vsphere-problem-detector openshift4/performance-addon-rhel8-operator quay/quay-container-security-operator-container quay/quay-container-security-operator-rhel8 quay/quay-operator-rhel8 rh-sso-7/sso74-openj9-openshift-rhel8 rhacm2/application-ui-rhel8 rhacm2/cert-manager-controller-rhel8 rhacm2/cert-policy-controller-rhel8 rhacm2/clusterlifecycle-state-metrics-rhel8 rhacm2/config-policy-controller-rhel8 rhacm2/console-api-rhel8 rhacm2/console-header-rhel8 rhacm2/console-rhel8 rhacm2/discovery-rhel8-operator rhacm2/endpoint-component-rhel8-operator rhacm2/endpoint-rhel8-operator rhacm2/governance-policy-propagator-rhel8 rhacm2/governance-policy-spec-sync-rhel8 rhacm2/governance-policy-status-sync-rhel8 rhacm2/governance-policy-template-sync-rhel8 rhacm2/grafana rhacm2/grc-ui-api-rhel8 rhacm2/grc-ui-rhel8 rhacm2/iam-policy-controller-rhel8 rhacm2/insights-client-rhel8 rhacm2/klusterlet-addon-controller-rhel8 rhacm2/klusterlet-addon-lease-controller-rhel8 rhacm2/klusterlet-addon-operator-rhel8 rhacm2/managedcluster-import-controller-rhel8 rhacm2/multicloud-manager-rhel8 rhacm2/multicluster-observability-rhel8-operator rhacm2/multicluster-operators-application-rhel8 rhacm2/multicluster-operators-channel-rhel8 rhacm2/multicluster-operators-deployable-rhel8 rhacm2/multicluster-operators-placementrule-rhel8 rhacm2/multicluster-operators-subscription-rhel8 rhacm2/multiclusterhub-repo-rhel8 rhacm2/multiclusterhub-rhel8 rhacm2/openshift-hive-rhel7 rhacm2/prometheus-alertmanager-rhel8 rhacm2/provider-credential-controller-rhel8 rhacm2/rcm-controller-rhel8 rhacm2/redisgraph-tls-rhel8 rhacm2/registration-rhel8 rhacm2/registration-rhel8-operator rhacm2/search-aggregator-rhel8 rhacm2/search-collector-rhel8 rhacm2/search-rhel8 rhacm2/search-ui-rhel8 rhacm2/submariner-addon-rhel8 rhacm2/thanos-receive-controller-rhel8 rhacm2/thanos-rhel7 rhacm2/work-rhel8 rhceph rhel7/couchbase-operator-admission rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/postgresql-96 rhel8/redis-5 rhmtc/openshift-migration-controller rhmtc/openshift-migration-operator rhmtc/openshift-migration-velero rhmtc/openshift-migration-velero-plugin-for-aws rhmtc/openshift-migration-velero-plugin-for-gcp rhmtc/openshift-migration-velero-plugin-for-microsoft-azure rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/postgresql-96-rhel7 rhscl/redis-32-rhel7 rook-ceph ubi8 ubi8/dotnet-50 ubi8/ruby-27 ubi8/ubi8-init volume-replication-operator
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 10.212495 2.710259 3.191318 2.235464 2.531879 2.065175 13.434725 0.060849 11.409653 4.269774 12.530345 11.946793 12.017999 14.544186 3.135820 10.098921 1.368805 1.368805 9.527233 8.325504 8.707050 3.956360 0.202113 4.190339 4.715256 2.861501 2.830902 9.552491 2.746994 4.209699 2.903732 4.027416 11.390301 4.320844 9.714880 11.357460 6.231020 4.247392 4.910821 4.072643 4.862510 3.841428 9.624172 9.684011 4.680520 4.174164 3.257445 5.148783 4.443662 4.088297 9.162374 4.755046 4.590602 3.561153 3.470536 9.267571 4.615115 8.124936 12.312973 9.722846 0.949997 2.201105 11.877791 1.870983 20.328555 9.881997 6.373029 1.516743 12.167505 4.502316 4.500797 2.664554 13.502305 2.982820 24.147302 18.252098 18.225759 21.819908 17.388136 21.049847 19.747568 24.991442 26.460856 22.792500 20.962237 24.094806 19.344642 20.762423 20.828972 17.541626 20.673653 20.941541 16.162125 14.562713 15.601577 19.836530 23.907342 24.002155 18.002954 16.637362 15.204368 16.201372 17.623032 27.170011 16.125317 17.217897 16.146708 17.840188 17.818970 19.392183 19.123314 19.021766 25.768520 26.437218 21.199134 24.401813 20.889458 19.339222 20.428393 20.989450 17.962332 16.313270 20.773993 22.447410 22.478224 22.108274 27.209110 22.564356 22.702737 26.857052 22.704280 23.594188 24.224838 34.405157 36.674626 39.959749 37.161722 31.414786 34.949369 38.338329 35.988386 34.023225 45.426438 34.027710 52.143121 41.365593 41.890508 44.016748 44.476493 47.210869 47.127143 50.912804 40.859017 52.385038 51.598048 44.405758 49.149815 50.436538 45.433999 46.793926 48.762990 39.468954 48.409586 42.533801 46.674942 49.515474 51.988255 46.457628 43.261360 39.732950 40.997311 49.057928 49.582509 38.516327 39.299219 47.292226 50.356002 33.982659 47.033300 38.498074 41.573013 40.355792 36.747398 36.361805 36.836134 34.458896 38.214497 35.099470 37.119567 30.150124 44.292937 39.592091 39.553169 40.084230 49.942175 36.843029 37.329289 40.542117 44.955204 44.111268 48.574556 46.307341 44.369794 45.910794 39.508773 34.606631 32.985475 34.639291 44.090552 39.612524 34.450975 57.224090 42.415056 39.005122 45.200547 42.256659 34.474696 35.060935 42.951406 37.308406 33.256690 32.525959 46.347208 46.179090 40.287774 35.113987 44.399612 36.195427 33.293605 43.686889 34.489515 43.517309 49.908296 45.100237 41.908622 38.367799 37.725580 35.023128 35.831688 36.258679 42.448970 43.681448 46.449438 38.668214 41.365216 36.328266 42.905825 43.751595 47.199185 39.250797 38.670660 39.065841 49.291122 49.675727 51.102242 49.866126 46.976162 42.999086 32.381362 34.385506 32.244991 31.699549 42.522840 33.865061 45.268921 35.058863 37.889456 47.310233 46.371312 46.942884 34.292780 37.672536 40.728524 33.813742 36.538439 25.383468 9.017498 9.496428 9.417449 17.938269 2.651199 5.940052 4.996048 7.138859 5.071356 4.613383 4.920721 4.546185 10.497612 10.183270 10.914794 6.399054 3.819182 4.031038 3.837636 1.796607 2.415324 2.121465 4.816271 4.371615 6.509100 8.354674 11.373735 7.786041 4.605525 12.552122 11.856307 12.207861 12.183611 12.771721 11.615330 5.183471 5.511970 21.912805 5.992666 5.595747 4.927704 2.486133 4.520238 11.231596 4.546019 4.326051 2.908384 2.733491 3.324565 6.268206 3.047492 3.137155 0.157851 11.084354 0.592422 0.600581 1.880347 1.880347 1.880347 1.035297 21.391517 26.015025 20.513775 17.389355 17.245405 15.674871 1.186782 1.026305 2.146571 2.146571 1.391054 1.008353 0.445479 0.689480 0.368605 0.541914 11.297479
00067111-ddb4-437a-93e3-3d1c8ac25d31 8.995419 2.160370 2.573723 1.748092 1.971441 1.568602 12.580306 0.081880 10.507684 5.253336 11.764190 11.057840 11.092143 13.459416 2.860462 8.831634 0.877811 0.877811 8.263298 7.337406 7.638742 3.668963 0.207943 4.165411 4.638435 2.784338 1.969992 8.729956 1.954378 3.406200 2.760756 4.057920 10.359551 4.260565 8.435964 10.297434 5.419165 4.246583 4.759925 3.849967 4.536060 3.731698 9.019578 9.184391 4.763916 4.110895 3.401647 5.105924 4.356395 4.081403 8.632997 4.069583 4.490968 3.465728 3.387500 8.719292 4.217066 7.389243 11.368609 8.255353 0.585342 2.146530 10.855041 1.608688 19.252776 8.639306 5.462955 1.329632 11.053145 3.974135 3.793639 2.821655 12.273794 2.119603 22.787701 16.634177 16.576359 19.810713 15.893566 19.477255 18.353301 22.945757 23.940164 21.356926 18.579887 22.277995 18.080111 19.302375 19.325830 16.142802 18.431724 19.196351 14.847969 13.410344 14.365913 18.459042 21.330494 22.212030 16.697619 15.605483 14.115977 15.083211 16.437457 25.350972 14.787583 16.131670 15.091690 16.592196 16.571977 18.054700 17.758342 17.664975 23.604029 24.262968 19.611136 22.377212 19.253856 17.839518 18.786249 19.411484 16.760895 15.226945 19.170626 21.012882 20.873241 20.695305 25.087056 21.298379 21.298379 24.904029 21.273650 21.067244 21.935807 32.173666 34.928318 37.496854 36.119021 30.053017 33.293612 35.987900 35.029250 34.585738 44.749798 36.431493 49.582004 38.828579 39.567380 40.552166 41.258028 43.357612 44.820004 46.363902 37.804790 48.171164 47.643612 40.829130 45.174347 46.704688 42.081066 42.624045 44.128002 36.409009 44.564838 40.328463 43.376584 45.527159 47.821542 42.935165 40.096420 36.903753 38.316941 44.441854 45.636397 35.361700 36.675670 44.221972 46.672996 32.248996 44.981493 36.767400 39.213371 38.274533 35.079603 34.245386 35.281542 32.896698 36.083885 32.857434 35.332774 28.759449 42.252413 37.614756 36.709702 38.068139 47.757868 35.819541 35.411510 37.920821 42.233175 40.679242 45.296822 44.089484 43.130386 48.779994 41.743431 37.244004 35.522157 36.623853 42.145308 35.305521 33.781576 50.273683 39.496913 36.694416 42.984737 40.084402 33.585049 33.423991 40.506689 35.174353 31.451876 31.006745 45.301014 43.368625 38.157342 33.374982 42.309380 34.652461 31.892913 41.186123 32.930283 40.652884 43.534651 41.648645 39.881168 36.464573 35.255268 33.181994 33.649506 35.062219 40.149992 41.310526 43.144867 37.313902 39.181932 35.668313 45.288156 37.776153 42.165237 35.014425 33.459647 35.137877 44.161834 46.869177 47.852550 46.709097 44.083185 41.030145 30.876338 32.894593 30.265035 30.083894 39.989919 32.255956 42.763205 33.240120 35.470148 45.036492 43.068913 44.571341 31.793308 35.746188 38.118174 31.935027 34.194315 23.510481 8.318964 8.366031 8.032382 16.378978 1.442449 4.597851 3.666086 5.502300 3.540183 3.208001 3.293483 2.741118 8.586992 8.544193 9.039980 4.872741 2.944516 2.983574 2.806419 0.731866 1.235324 0.665272 3.494510 3.019229 5.037632 6.833176 9.321538 6.229780 2.869374 10.420578 9.692549 9.978749 9.972624 10.566197 9.440567 3.560109 3.725685 19.884362 3.619925 4.412654 3.385024 1.050930 2.029219 8.465909 3.116701 2.998431 1.417360 1.376821 2.241953 4.794953 1.508559 0.978321 0.033841 9.998611 0.078088 0.104292 1.363427 1.363427 1.363427 0.420853 20.195932 24.297575 19.305067 16.271568 16.099687 14.695559 0.688198 0.482011 1.583662 1.583662 0.755498 0.865163 0.383049 0.869834 0.368129 0.808570 10.370895
002663ad-bcf4-4c7c-9530-ecb351fe4001 0.234350 0.054241 0.061423 0.042898 0.056661 0.046651 0.320669 0.000000 0.230667 0.042898 0.210621 0.232727 0.254084 0.286222 0.101032 0.195826 0.030077 0.030077 0.217672 0.167647 0.158732 0.076878 0.000000 0.131354 0.069532 0.073358 0.030154 0.186912 0.030549 0.029199 0.044547 0.085325 0.153505 0.037568 0.146624 0.153505 0.045733 0.034117 0.037187 0.031276 0.058078 0.024501 0.159012 0.172721 0.066841 0.037073 0.031015 0.055430 0.042787 0.051985 0.162318 0.038389 0.040735 0.042285 0.050316 0.166624 0.032285 0.132413 0.260545 0.195989 0.015446 0.021565 0.241863 0.026759 0.281851 0.190426 0.069712 0.015096 0.265217 0.051866 0.055640 0.083305 0.301221 0.049249 0.296759 0.246576 0.241439 0.349771 0.232268 0.294837 0.279087 0.384531 0.413846 0.407220 0.276864 0.438163 0.298218 0.307715 0.307715 0.195443 0.285582 0.326840 0.184702 0.190683 0.193719 0.229789 0.336268 0.390845 0.213756 0.234497 0.209352 0.221808 0.229036 0.435769 0.182712 0.208265 0.236433 0.229964 0.239159 0.251761 0.220867 0.248288 0.408600 0.420555 0.262308 0.391023 0.291916 0.277669 0.292920 0.241622 0.238789 0.216620 0.269894 0.312689 0.307712 0.304105 0.451813 0.317830 0.317830 0.446130 0.317830 0.282132 0.295533 0.590296 0.513729 0.619681 0.463670 0.422792 0.578947 0.670088 0.448633 0.458749 0.780148 0.495614 0.642073 0.514902 0.697794 0.601349 0.584889 0.682043 0.685380 0.674267 0.525038 0.721489 0.700118 0.685874 0.662047 0.607593 0.566363 0.594640 0.615306 0.682507 0.710667 0.490223 0.653217 0.670996 0.713805 0.631984 0.549523 0.517423 0.461941 0.678058 1.024813 0.540270 0.529198 0.584343 0.796774 0.421619 0.603851 0.507706 0.637253 0.529392 0.531015 0.504174 0.532917 0.539758 0.492350 0.502762 0.477512 0.408955 0.659091 0.597568 0.585768 0.492350 0.626597 0.463670 0.523331 0.630467 0.603851 0.546033 0.596825 0.739541 0.644425 0.597780 0.574783 0.501428 0.478933 0.554271 0.584343 0.471574 0.516099 0.511229 0.537195 0.793437 0.559204 0.525297 0.473389 1.610294 1.610294 1.314442 1.036811 0.536484 0.721489 0.681475 0.554418 0.440920 0.547429 0.455515 0.415610 0.607842 0.457610 0.560314 0.554039 0.604380 0.567612 0.526552 0.551601 0.507822 0.563376 0.485654 0.715503 0.694368 0.747661 0.549571 0.650263 0.462070 0.538065 0.517396 0.547429 0.527192 0.454530 0.483317 0.710227 0.642073 0.824695 0.664003 0.764381 0.577685 0.446154 0.466508 0.434539 0.447353 0.696360 0.472658 0.605818 0.520000 0.604116 0.632500 0.597780 0.632500 0.596936 0.529198 0.634570 0.482165 0.525555 0.421226 0.156667 0.150955 0.177790 0.256597 0.015931 0.042232 0.009368 0.064977 0.009112 0.022286 0.032002 0.023883 0.198093 0.168402 0.196865 0.076086 0.035478 0.047868 0.038013 0.013290 0.000000 0.000000 0.009502 0.020889 0.017583 0.049691 0.151667 0.054605 0.055813 0.160126 0.143092 0.150955 0.158741 0.148431 0.141253 0.019371 0.020889 0.274649 0.051866 0.042828 0.010555 0.026596 0.038740 0.195760 0.067615 0.041561 0.012086 0.011462 0.015359 0.027212 0.012086 0.012540 0.000000 0.201006 0.000000 0.013290 0.036700 0.036700 0.036700 0.020408 0.277572 0.439695 0.277215 0.231754 0.231754 0.208186 0.000000 0.011870 0.033060 0.033060 0.017857 0.000000 0.000000 0.000000 0.000000 0.000000 0.220283
00351e6e-53ce-465e-9493-cf0cd2367049 10.212495 2.710259 3.191318 2.235464 2.531879 2.065175 13.434725 0.060849 11.409653 4.269774 12.530345 11.946793 12.017999 14.544186 3.135820 10.098921 1.368805 1.368805 9.527233 8.325504 8.707050 3.956360 0.202113 4.190339 4.715256 2.861501 2.830902 9.552491 2.746994 4.209699 2.903732 4.027416 11.390301 4.320844 9.714880 11.357460 6.231020 4.247392 4.910821 4.072643 4.862510 3.841428 9.624172 9.684011 4.680520 4.174164 3.257445 5.148783 4.443662 4.088297 9.162374 4.755046 4.590602 3.561153 3.470536 9.267571 4.615115 8.124936 12.312973 9.722846 0.949997 2.201105 11.877791 1.870983 20.328555 9.881997 6.373029 1.516743 12.167505 4.502316 4.500797 2.664554 13.502305 2.982820 24.147302 18.252098 18.225759 21.819908 17.388136 21.049847 19.747568 24.991442 26.460856 22.792500 20.962237 24.094806 19.344642 20.762423 20.828972 17.541626 20.673653 20.941541 16.162125 14.562713 15.601577 19.836530 23.907342 24.002155 18.002954 16.637362 15.204368 16.201372 17.623032 27.170011 16.125317 17.217897 16.146708 17.840188 17.818970 19.392183 19.123314 19.021766 25.768520 26.437218 21.199134 24.401813 20.889458 19.339222 20.428393 20.989450 17.962332 16.313270 20.773993 22.447410 22.478224 22.108274 27.209110 22.564356 22.702737 26.857052 22.704280 23.594188 24.224838 34.405157 36.674626 39.959749 37.161722 31.414786 34.949369 38.338329 35.988386 34.023225 45.426438 34.027710 52.143121 41.365593 41.890508 44.016748 44.476493 47.210869 47.127143 50.912804 40.859017 52.385038 51.598048 44.405758 49.149815 50.436538 45.433999 46.793926 48.762990 39.468954 48.409586 42.533801 46.674942 49.515474 51.988255 46.457628 43.261360 39.732950 40.997311 49.057928 49.582509 38.516327 39.299219 47.292226 50.356002 33.982659 47.033300 38.498074 41.573013 40.355792 36.747398 36.361805 36.836134 34.458896 38.214497 35.099470 37.119567 30.150124 44.292937 39.592091 39.553169 40.084230 49.942175 36.843029 37.329289 40.542117 44.955204 44.111268 48.574556 46.307341 44.369794 45.910794 39.508773 34.606631 32.985475 34.639291 44.090552 39.612524 34.450975 57.224090 42.415056 39.005122 45.200547 42.256659 34.474696 35.060935 42.951406 37.308406 33.256690 32.525959 46.347208 46.179090 40.287774 35.113987 44.399612 36.195427 33.293605 43.686889 34.489515 43.517309 49.908296 45.100237 41.908622 38.367799 37.725580 35.023128 35.831688 36.258679 42.448970 43.681448 46.449438 38.668214 41.365216 36.328266 42.905825 43.751595 47.199185 39.250797 38.670660 39.065841 49.291122 49.675727 51.102242 49.866126 46.976162 42.999086 32.381362 34.385506 32.244991 31.699549 42.522840 33.865061 45.268921 35.058863 37.889456 47.310233 46.371312 46.942884 34.292780 37.672536 40.728524 33.813742 36.538439 25.383468 9.017498 9.496428 9.417449 17.938269 2.651199 5.940052 4.996048 7.138859 5.071356 4.613383 4.920721 4.546185 10.497612 10.183270 10.914794 6.399054 3.819182 4.031038 3.837636 1.796607 2.415324 2.121465 4.816271 4.371615 6.509100 8.354674 11.373735 7.786041 4.605525 12.552122 11.856307 12.207861 12.183611 12.771721 11.615330 5.183471 5.511970 21.912805 5.992666 5.595747 4.927704 2.486133 4.520238 11.231596 4.546019 4.326051 2.908384 2.733491 3.324565 6.268206 3.047492 3.137155 0.157851 11.084354 0.592422 0.600581 1.880347 1.880347 1.880347 1.035297 21.391517 26.015025 20.513775 17.389355 17.245405 15.674871 1.186782 1.026305 2.146571 2.146571 1.391054 1.008353 0.445479 0.689480 0.368605 0.541914 11.297479
003ba133-e754-4d5a-bc57-675b386d1f05 10.951183 3.517931 3.740535 2.764705 3.133221 2.630070 13.867011 0.074160 11.707567 4.515827 13.604715 12.251898 12.340407 15.047085 3.314593 10.975715 1.810429 1.810429 10.264170 9.049425 9.321219 4.135661 0.228689 4.404162 5.666622 2.978878 3.613465 9.961682 3.475348 5.205738 3.398058 4.405590 12.276364 4.802197 10.619164 12.072765 7.167730 4.685000 5.896319 4.481388 5.395678 4.220439 10.185902 10.232341 5.134309 4.642546 3.672282 5.561682 5.326936 4.535108 9.727619 5.142252 5.523034 4.045146 3.905986 9.836355 6.088212 9.173560 12.815071 10.692749 0.976592 2.355708 12.274347 1.993156 20.504672 10.643172 6.634164 1.645692 12.564222 5.012691 4.914112 2.793988 13.907210 3.106547 24.666830 18.948801 18.917567 22.563003 18.127229 21.791420 20.544236 25.679524 27.554210 23.393518 21.829677 25.029697 20.188353 21.559944 21.635540 19.006940 21.501535 21.673315 17.536614 15.192013 16.202822 21.504940 24.818715 24.809079 19.546032 17.491144 15.910928 16.878435 18.269513 27.892956 17.526460 18.767355 17.091931 18.709167 18.810325 20.258861 20.783715 19.858484 26.708778 27.415513 22.670968 25.324686 21.937058 20.284681 21.514110 22.805650 18.783911 17.055797 21.668875 23.412183 23.332240 23.069336 28.165647 23.199662 23.572612 27.776278 23.554172 24.471570 25.082704 34.999842 37.068444 40.653670 38.267905 31.762651 35.321064 38.997088 37.063366 35.065237 46.057781 34.380187 52.408816 41.112126 42.478394 45.057817 45.258612 48.193247 48.041676 52.029166 42.315533 53.341091 52.582388 45.313231 50.084954 51.302536 46.876742 47.742788 49.787969 40.332506 49.348652 43.309763 47.633845 50.639522 52.979576 47.352215 44.090043 41.085790 42.753519 50.035337 50.534582 39.662977 39.914909 47.407350 50.998583 34.684692 47.372633 38.970027 42.270643 40.821859 37.245321 36.885897 37.286145 34.886896 38.552029 35.662371 38.208289 30.552324 44.649399 40.040376 40.123161 40.782520 50.280675 37.932309 37.742899 41.255060 45.276380 44.612008 48.932161 46.929996 44.929260 46.326396 40.018337 35.044324 33.439736 35.116443 44.368703 40.129647 34.817999 57.623213 42.894928 39.589656 45.779836 43.442380 35.558426 35.480820 43.595959 37.918530 33.801034 32.965421 46.975573 46.998235 40.674312 36.312921 44.984013 36.852530 33.841841 44.183583 34.969849 43.904332 50.260364 45.534900 42.334941 39.538761 38.200550 35.546339 36.479266 37.335765 43.097369 44.329016 47.209978 39.109179 42.098739 37.440905 43.372133 44.129824 47.623027 39.762595 39.376851 39.549285 50.019401 50.014459 51.795133 50.313201 47.741496 44.458119 33.902644 35.937505 33.335965 32.635994 43.995921 34.841518 45.766723 35.516435 38.585127 47.623246 46.716512 47.282906 35.411800 38.110054 41.442377 34.359862 37.070359 26.527273 9.672069 10.774288 10.534953 18.773565 4.293197 8.756880 8.464677 8.965426 8.703528 6.355292 6.973385 6.391876 12.163150 11.749437 12.553777 8.576116 6.221683 6.194553 5.877374 2.715820 3.963806 3.757729 8.498035 5.918440 11.066506 13.482184 14.596288 10.617950 6.397407 14.376132 14.049615 14.339744 14.231061 14.857704 14.110938 7.068935 7.531914 23.223087 7.734944 8.075715 8.152159 3.923503 6.274726 12.917456 6.465550 7.028419 4.865130 4.573675 5.388398 9.007374 4.337316 4.343723 0.318409 11.972073 1.114720 1.121995 2.411446 2.411446 2.411446 1.565021 22.602342 26.656079 21.071705 17.942118 17.793542 16.191431 1.735951 1.551169 2.678562 2.678562 1.915918 1.144896 0.504067 0.806203 0.394792 0.592381 11.895507
X_cont.shape
(1881, 353)

1881 rows corresponds to 1881 cluster_id.

3D plot for image_layer_dataset

pca3 = PCA(n_components=3)
pc3 = pca3.fit_transform(X_image)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pc3[:, 0],
    ys=pc3[:, 1],
    zs=pc3[:, 2],
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.title("PCA for image_layers dataset (3D)")
plt.show()
../_images/ML_clustering_118_0.png

3D interactive visualization for image dataset

fig = px.scatter_3d(pc3, x=pc3[:, 0], y=pc3[:, 1], z=pc3[:, 2], opacity=0.5)
fig.show()

3D principal component analysis for container dataset

pcc3 = pca3.fit_transform(X_cont)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pcc3[:, 0],
    ys=pcc3[:, 1],
    zs=pcc3[:, 2],
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.title("PCA for containers dataset (3D)")
plt.show()
../_images/ML_clustering_123_0.png

3D interactive visualization

fig = px.scatter_3d(pcc3, x=pcc3[:, 0], y=pcc3[:, 1], z=pcc3[:, 2], opacity=1)
fig.show()

Now that we have visualized both image_layer dataset and container dataset in both 2D and 3D. We now implement Kmeans to identify different clusters in the set.

Before applying KMeans by inserting the number of clusters, we first find the optimal number of clusters in the dataset by using the Elbow method.

The Elbow Method

Applying the elbow method for the image layer dataset, we set the range of cluster from 1 - 10.

# How to find the best number of clusters?

no_of_clusters = range(1, 10)
inertia = []

for f in tqdm(no_of_clusters):
    kmeans = KMeans(n_clusters=f, random_state=2)
    kmeans = kmeans.fit(X_image)
    u = kmeans.inertia_
    inertia.append(u)
100%|██████████| 9/9 [00:06<00:00,  1.34it/s]

Inertia vs Number of cluster plot

fig, (ax1) = plt.subplots(1, figsize=(16, 6))
xx = np.arange(len(no_of_clusters))
ax1.plot(xx, inertia)
ax1.set_xticks(xx)
ax1.set_xticklabels(no_of_clusters, rotation="horizontal")
plt.xlabel("Number of clusters")
plt.ylabel("SSE")
plt.title("Inertia plot per K")
plt.show()
../_images/ML_clustering_129_0.png

We see that the optimal number of clusters for image_layer dataset is 5.

Now checking for the containers dataset.

# How to find the best number of clusters?

no_of_clusters = range(1, 10)
inertia = []

for f in tqdm(no_of_clusters):
    kmeans = KMeans(n_clusters=f, random_state=2)
    kmeans = kmeans.fit(X_cont)
    u = kmeans.inertia_
    inertia.append(u)
100%|██████████| 9/9 [00:19<00:00,  2.19s/it]
fig, (ax1) = plt.subplots(1, figsize=(16, 6))
xx = np.arange(len(no_of_clusters))
ax1.plot(xx, inertia)
ax1.set_xticks(xx)
ax1.set_xticklabels(no_of_clusters, rotation="horizontal")
plt.xlabel("Number of clusters")
plt.ylabel("SSE")
plt.title("Inertia plot per K")
Text(0.5, 1.0, 'Inertia plot per K')
../_images/ML_clustering_132_1.png

Optimal number of clusters for container dataset: 3

Implementing KMeans for image_layer_dataset

We take the number of clusters = 5 and apply the KMeans algorithm.

kmeans = KMeans(n_clusters=5, random_state=2)
kmeans = kmeans.fit(X_image)
clusters = kmeans.predict(X_image)
clusters
array([0, 0, 3, ..., 0, 0, 0], dtype=int32)
cluster_image_df = X_image
cluster_image_df["clusters"] = clusters
cluster_image_df.head()
3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 jboss-amq-6/amq63-openshift openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-grafana openshift/ose-jenkins openshift/ose-must-gather openshift/ose-oauth-proxy openshift/ose-tests openshift/ose-tools rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/redis-5 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/redis-32-rhel7 ubi8/dotnet-50 ubi8/ruby-27 clusters
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 0.124094 0.292594 0.177182 0.147851 0.154433 2.684535 10.221675 10.190476 6.483450 6.483450 5.695925 8.241379 7.128506 6.952486 0.0 0.0 0.121059 0.121059 0.0 0.084199 0.088312 0.185413 0.088312 0.0 0.0 0
00351e6e-53ce-465e-9493-cf0cd2367049 0.124094 0.292594 0.177182 0.147851 0.154433 2.684535 10.221675 10.190476 6.483450 6.483450 5.695925 8.241379 7.128506 6.952486 0.0 0.0 0.121059 0.121059 0.0 0.084199 0.088312 0.185413 0.088312 0.0 0.0 0
003ba133-e754-4d5a-bc57-675b386d1f05 0.056678 0.134699 0.063974 0.068381 0.071675 1.691729 6.793103 4.190476 4.175758 4.175758 3.695925 6.241379 4.503506 4.503506 0.0 0.0 0.077263 0.077263 0.0 0.043103 0.045455 0.107491 0.045455 0.0 0.0 3
00479ead-b7fc-49c2-ae20-3990a9b3d08c 0.124094 0.292594 0.177182 0.147851 0.154433 2.684535 10.221675 10.190476 6.483450 6.483450 5.695925 8.241379 7.128506 6.952486 0.0 0.0 0.121059 0.121059 0.0 0.084199 0.088312 0.185413 0.088312 0.0 0.0 0
00748c32-15c3-4586-98fb-e5078ca9f3b8 0.124094 0.292594 0.177182 0.147851 0.154433 2.684535 10.221675 10.190476 6.483450 6.483450 5.695925 8.241379 7.128506 6.952486 0.0 0.0 0.121059 0.121059 0.0 0.084199 0.088312 0.185413 0.088312 0.0 0.0 0

Visualizing the results of the 3D PCA

pca_i = pca3.fit_transform(cluster_image_df)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pca_i[:, 0], ys=pca_i[:, 1], zs=pca_i[:, 2], c=cluster_image_df["clusters"]
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.show()
../_images/ML_clustering_141_0.png

3D interactive visualization for image dataset

fig = px.scatter_3d(
    pca_i,
    x=pca_i[:, 0],
    y=pca_i[:, 1],
    z=pca_i[:, 2],
    color=cluster_image_df["clusters"],
    opacity=0.1,
)
fig.show()

Listing out the groups of image_name in different clusters

df1 = pd.DataFrame([])
global_mean = cluster_image_df.mean().sort_values(ascending=False)


for c in tqdm(range(0, 5)):
    c = cluster_image_df[cluster_image_df.clusters == c].iloc[:, :-1].mean()
    diff = c - global_mean
    d = diff.sort_values(ascending=False).head(20)
    d1 = pd.DataFrame(d).reset_index()
    d1.rename(columns={"index": "image_name"}, inplace=True)
    df1 = df1.append(d1.image_name)
df1 = df1.reset_index()
df1.rename(
    index={
        0: "cluster 1",
        1: "cluster 2",
        2: "cluster 3",
        3: "cluster 4",
        4: "cluster 5",
    },
    inplace=True,
)
df1.drop(columns="index", inplace=True)

df1
100%|██████████| 5/5 [00:00<00:00, 216.83it/s]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
cluster 1 openshift/ose-cli-artifacts ubi8/dotnet-50 ubi8/ruby-27 jboss-amq-6/amq63-openshift openshift/ose-oauth-proxy 3scale-amp2/memcached-rhel7 openshift/ose-cli 3scale-amp2/backend-rhel7 3scale-amp2/zync-rhel7 rhscl/mongodb-36-rhel7 3scale-amp2/system-rhel7 rhscl/redis-32-rhel7 3scale-amp2/apicast-gateway-rhel8 openshift/ose-grafana rhscl/mysql-57-rhel7 openshift/ose-jenkins rhel8/httpd-24 rhel8/redis-5 openshift/ose-tests rhel8/mysql-80
cluster 2 openshift/ose-tools openshift/ose-must-gather openshift/ose-tests openshift/ose-cli openshift/ose-jenkins openshift/ose-grafana openshift/ose-oauth-proxy openshift/ose-cli-artifacts jboss-amq-6/amq63-openshift rhel8/postgresql-12 rhel8/postgresql-10 3scale-amp2/apicast-gateway-rhel8 rhel8/mysql-80 rhel8/httpd-24 rhscl/postgresql-10-rhel7 3scale-amp2/backend-rhel7 rhel8/redis-5 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 rhscl/mysql-57-rhel7
cluster 3 rhel8/postgresql-12 rhel8/postgresql-10 rhscl/postgresql-10-rhel7 rhel8/mysql-80 rhel8/redis-5 rhel8/httpd-24 rhscl/mysql-57-rhel7 rhscl/redis-32-rhel7 rhscl/mongodb-36-rhel7 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 openshift/ose-tests openshift/ose-must-gather openshift/ose-grafana ubi8/ruby-27 ubi8/dotnet-50 openshift/ose-cli-artifacts
cluster 4 ubi8/ruby-27 ubi8/dotnet-50 rhel8/mysql-80 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhel8/redis-5 rhel8/httpd-24 rhscl/redis-32-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 3scale-amp2/apicast-gateway-rhel8 rhscl/postgresql-10-rhel7 3scale-amp2/memcached-rhel7 rhel8/postgresql-10 rhel8/postgresql-12 3scale-amp2/backend-rhel7 jboss-amq-6/amq63-openshift openshift/ose-must-gather openshift/ose-grafana openshift/ose-jenkins
cluster 5 3scale-amp2/zync-rhel7 3scale-amp2/system-rhel7 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/apicast-gateway-rhel8 rhscl/mysql-57-rhel7 rhscl/redis-32-rhel7 rhscl/postgresql-10-rhel7 rhscl/mongodb-36-rhel7 jboss-amq-6/amq63-openshift rhel8/postgresql-10 rhel8/postgresql-12 rhel8/mysql-80 rhel8/redis-5 rhel8/httpd-24 openshift/ose-must-gather openshift/ose-tests openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-grafana

We generated the above dataframe by differencitating the global mean with the mean of the individual cluster, represented as a index in the dataframe. Each row in the dataframe df1, represents groups of top 20 image_names having a higher probability of occuring/running for the same user.

Implementing KMeans for container dataset

We take the number of clusters = 3. Because of the large number of unique features, the program will be computationally intensive. We apply KMeans algorithm by selecting 10 clusters.

kmeans = KMeans(n_clusters=3, random_state=2)
kmeans = kmeans.fit(X_cont)
clusters = kmeans.predict(X_cont)
clusters
array([1, 1, 0, ..., 1, 1, 2], dtype=int32)
cluster_cont_df = X_cont
cluster_cont_df["clusters"] = clusters
cluster_cont_df.head()
3scale-amp2/3scale-rhel7-operator 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 Calico Operator Cilium Cloud Native PostgreSQL Operator Elastic Cloud on Kubernetes F5 BIG-IP Controller Operator NGINX Ingress Operator NVIDIA GPU Operator Seldon Operator alertmanager amq7/amq-broker-rhel7-operator amq7/amq-streams-kafka-25-rhel7 amq7/amq-streams-kafka-26-rhel7 amq7/amq-streams-rhel7-operator amq7/amq-streams-rhel7-operator-metadata ansible-automation-platform/platform-resource-rhel7-operator ansible-tower cephcsi cert-manager cainjector cert-manager controller cert-manager webhook codeready-workspaces/devfileregistry-rhel8 codeready-workspaces/operator codeready-workspaces/pluginregistry-rhel8 codeready-workspaces/server-rhel8 collectd-exporter container-native-virtualization/bridge-marker container-native-virtualization/cluster-network-addons-operator container-native-virtualization/cnv-containernetworking-plugins container-native-virtualization/hostpath-provisioner-rhel8-operator container-native-virtualization/hyperconverged-cluster-operator container-native-virtualization/hyperconverged-cluster-webhook-rhel8 container-native-virtualization/kubemacpool container-native-virtualization/kubernetes-nmstate-handler-rhel8 container-native-virtualization/kubevirt-cpu-model-nfd-plugin container-native-virtualization/kubevirt-cpu-node-labeller container-native-virtualization/kubevirt-kvm-info-nfd-plugin container-native-virtualization/kubevirt-ssp-operator container-native-virtualization/node-maintenance-operator container-native-virtualization/ovs-cni-marker container-native-virtualization/ovs-cni-plugin container-native-virtualization/virt-api container-native-virtualization/virt-cdi-apiserver container-native-virtualization/virt-cdi-controller container-native-virtualization/virt-cdi-importer container-native-virtualization/virt-cdi-operator container-native-virtualization/virt-cdi-uploadproxy container-native-virtualization/virt-controller container-native-virtualization/virt-handler container-native-virtualization/virt-launcher container-native-virtualization/virt-operator container-native-virtualization/vm-import-controller-rhel8 container-native-virtualization/vm-import-operator-rhel8 costmanagement-metrics-operator distributed-tracing/jaeger-rhel8-operator grafana ibm common service webhook ibm-events-operator ibm-postgresql jboss-amq-6/amq63-openshift jboss-eap-7/eap73-rhel8-operator kube-state-metrics mcg-core mcg-operator mcr.microsoft.com/mssql/rhel8/server must-gather-service ocp-tools-4/odo-init-image ocs-operator ocs-registry openshift-compliance-content openshift-gitops-1-tech-preview/argocd-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8-operator openshift-gitops-1-tech-preview/kam-delivery-rhel8 openshift-gitops-1/gitops-rhel8 openshift-gitops-1/kam-delivery-rhel8 openshift-logging/cluster-logging-operator-bundle openshift-logging/cluster-logging-rhel8-operator openshift-logging/elasticsearch-operator-bundle openshift-logging/elasticsearch-proxy-rhel8 openshift-logging/elasticsearch-rhel8-operator openshift-logging/elasticsearch6-rhel8 openshift-logging/fluentd-rhel8 openshift-logging/kibana6-rhel8 openshift-pipelines-tech-preview/pipelines-controller-rhel8 openshift-pipelines-tech-preview/pipelines-operator-proxy-rhel8 openshift-pipelines-tech-preview/pipelines-rhel8-operator openshift-pipelines-tech-preview/pipelines-triggers-controller-rhel8 openshift-pipelines-tech-preview/pipelines-triggers-webhook-rhel8 openshift-pipelines-tech-preview/pipelines-webhook-rhel8 openshift-pipelines/pipelines-controller-rhel8 openshift-pipelines/pipelines-operator-proxy-rhel8 openshift-pipelines/pipelines-rhel8-operator openshift-pipelines/pipelines-triggers-controller-rhel8 openshift-pipelines/pipelines-triggers-core-interceptors-rhel8 openshift-pipelines/pipelines-triggers-eventlistenersink-rhel8 openshift-pipelines/pipelines-triggers-webhook-rhel8 openshift-pipelines/pipelines-webhook-rhel8 openshift-sandboxed-containers-operator openshift-serverless-1-tech-preview/eventing-kafka-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-dispatcher-rhel8 openshift-serverless-1/eventing-mtbroker-ingress-rhel8 openshift-serverless-1/eventing-mtchannel-broker-rhel8 openshift-serverless-1/eventing-mtping-rhel8 openshift-serverless-1/eventing-sugar-controller-rhel8 openshift-serverless-1/eventing-webhook-rhel8 openshift-serverless-1/ingress-rhel8-operator openshift-serverless-1/knative-rhel8-operator openshift-serverless-1/kourier-control-rhel8 openshift-serverless-1/serverless-rhel8-operator openshift-serverless-1/serving-activator-rhel8 openshift-serverless-1/serving-autoscaler-hpa-rhel8 openshift-serverless-1/serving-autoscaler-rhel8 openshift-serverless-1/serving-controller-rhel8 openshift-serverless-1/serving-domain-mapping-rhel8 openshift-serverless-1/serving-domain-mapping-webhook-rhel8 openshift-serverless-1/serving-queue-rhel8 openshift-service-mesh/galley-rhel8 openshift-service-mesh/grafana-rhel8 openshift-service-mesh/istio-cni-rhel8 openshift-service-mesh/istio-rhel8-operator openshift-service-mesh/kiali-rhel7 openshift-service-mesh/kiali-rhel8 openshift-service-mesh/kiali-rhel8-operator openshift-service-mesh/pilot-rhel8 openshift-service-mesh/prometheus-rhel8 openshift-service-mesh/proxyv2-rhel8 openshift/compliance-operator openshift/ose-aws-ebs-csi-driver openshift/ose-aws-ebs-csi-driver-operator openshift/ose-aws-machine-controllers openshift/ose-aws-pod-identity-webhook openshift/ose-azure-disk-csi-driver openshift/ose-azure-disk-csi-driver-operator openshift/ose-azure-machine-controllers openshift/ose-baremetal-machine-controllers openshift/ose-baremetal-operator openshift/ose-baremetal-runtimecfg openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-cloud-credential-operator openshift/ose-cluster-authentication-operator openshift/ose-cluster-autoscaler openshift/ose-cluster-autoscaler-operator openshift/ose-cluster-baremetal-operator openshift/ose-cluster-config-operator openshift/ose-cluster-csi-snapshot-controller-operator openshift/ose-cluster-dns-operator openshift/ose-cluster-etcd-operator openshift/ose-cluster-image-registry-operator openshift/ose-cluster-ingress-operator openshift/ose-cluster-kube-apiserver-operator openshift/ose-cluster-kube-controller-manager-operator openshift/ose-cluster-kube-descheduler-operator openshift/ose-cluster-kube-scheduler-operator openshift/ose-cluster-kube-storage-version-migrator-operator openshift/ose-cluster-logging-operator openshift/ose-cluster-machine-approver openshift/ose-cluster-monitoring-operator openshift/ose-cluster-network-operator openshift/ose-cluster-nfd-operator openshift/ose-cluster-node-tuning-operator openshift/ose-cluster-openshift-apiserver-operator openshift/ose-cluster-openshift-controller-manager-operator openshift/ose-cluster-policy-controller openshift/ose-cluster-samples-operator openshift/ose-cluster-storage-operator openshift/ose-clusterresourceoverride-rhel8-operator openshift/ose-configmap-reloader openshift/ose-console openshift/ose-console-operator openshift/ose-container-networking-plugins openshift/ose-coredns openshift/ose-csi-driver-manila openshift/ose-csi-driver-manila-operator openshift/ose-csi-driver-nfs openshift/ose-csi-external-attacher openshift/ose-csi-external-provisioner openshift/ose-csi-external-resizer openshift/ose-csi-external-snapshotter openshift/ose-csi-livenessprobe openshift/ose-csi-node-driver-registrar openshift/ose-csi-snapshot-controller openshift/ose-csi-snapshot-validation-webhook openshift/ose-deployer openshift/ose-docker-builder openshift/ose-docker-registry openshift/ose-egress-router-cni openshift/ose-etcd openshift/ose-gcp-machine-controllers openshift/ose-gcp-pd-csi-driver openshift/ose-gcp-pd-csi-driver-operator openshift/ose-grafana openshift/ose-haproxy-router openshift/ose-hyperkube openshift/ose-insights-operator openshift/ose-installer openshift/ose-ironic openshift/ose-ironic-inspector openshift/ose-ironic-ipa-downloader openshift/ose-ironic-machine-os-downloader openshift/ose-ironic-static-ip-manager openshift/ose-jenkins openshift/ose-k8s-prometheus-adapter openshift/ose-keepalived-ipfailover openshift/ose-kube-rbac-proxy openshift/ose-kube-state-metrics openshift/ose-kube-storage-version-migrator openshift/ose-kuryr-cni openshift/ose-kuryr-controller openshift/ose-libvirt-machine-controllers openshift/ose-local-storage-diskmaker openshift/ose-local-storage-operator openshift/ose-local-storage-operator-bundle openshift/ose-local-storage-static-provisioner openshift/ose-logging-elasticsearch6 openshift/ose-machine-api-operator openshift/ose-machine-config-operator openshift/ose-mdns-publisher openshift/ose-multus-admission-controller openshift/ose-multus-cni openshift/ose-multus-route-override-cni openshift/ose-multus-whereabouts-ipam-cni openshift/ose-must-gather openshift/ose-network-metrics-daemon openshift/ose-oauth-apiserver openshift/ose-oauth-proxy openshift/ose-oauth-server openshift/ose-openshift-apiserver openshift/ose-openshift-controller-manager openshift/ose-openshift-state-metrics openshift/ose-openstack-cinder-csi-driver openshift/ose-openstack-cinder-csi-driver-operator openshift/ose-openstack-machine-controllers openshift/ose-operator-lifecycle-manager openshift/ose-operator-marketplace openshift/ose-operator-registry openshift/ose-ovirt-csi-driver openshift/ose-ovirt-csi-driver-operator openshift/ose-ovirt-machine-controllers openshift/ose-ovn-kubernetes openshift/ose-prom-label-proxy openshift/ose-prometheus openshift/ose-prometheus-alertmanager openshift/ose-prometheus-config-reloader openshift/ose-prometheus-node-exporter openshift/ose-prometheus-operator openshift/ose-ptp openshift/ose-ptp-operator openshift/ose-sdn openshift/ose-service-ca-operator openshift/ose-sriov-cni openshift/ose-sriov-dp-admission-controller openshift/ose-sriov-infiniband-cni openshift/ose-sriov-network-config-daemon openshift/ose-sriov-network-device-plugin openshift/ose-sriov-network-operator openshift/ose-sriov-network-webhook openshift/ose-telemeter openshift/ose-template-service-broker openshift/ose-template-service-broker-operator openshift/ose-tests openshift/ose-thanos openshift/ose-tools openshift/ose-vertical-pod-autoscaler-rhel8-operator openshift/ose-vsphere-csi-driver openshift/ose-vsphere-csi-driver-operator openshift/ose-vsphere-csi-driver-syncer openshift/ose-vsphere-problem-detector openshift4/performance-addon-rhel8-operator quay/quay-container-security-operator-container quay/quay-container-security-operator-rhel8 quay/quay-operator-rhel8 rh-sso-7/sso74-openj9-openshift-rhel8 rhacm2/application-ui-rhel8 rhacm2/cert-manager-controller-rhel8 rhacm2/cert-policy-controller-rhel8 rhacm2/clusterlifecycle-state-metrics-rhel8 rhacm2/config-policy-controller-rhel8 rhacm2/console-api-rhel8 rhacm2/console-header-rhel8 rhacm2/console-rhel8 rhacm2/discovery-rhel8-operator rhacm2/endpoint-component-rhel8-operator rhacm2/endpoint-rhel8-operator rhacm2/governance-policy-propagator-rhel8 rhacm2/governance-policy-spec-sync-rhel8 rhacm2/governance-policy-status-sync-rhel8 rhacm2/governance-policy-template-sync-rhel8 rhacm2/grafana rhacm2/grc-ui-api-rhel8 rhacm2/grc-ui-rhel8 rhacm2/iam-policy-controller-rhel8 rhacm2/insights-client-rhel8 rhacm2/klusterlet-addon-controller-rhel8 rhacm2/klusterlet-addon-lease-controller-rhel8 rhacm2/klusterlet-addon-operator-rhel8 rhacm2/managedcluster-import-controller-rhel8 rhacm2/multicloud-manager-rhel8 rhacm2/multicluster-observability-rhel8-operator rhacm2/multicluster-operators-application-rhel8 rhacm2/multicluster-operators-channel-rhel8 rhacm2/multicluster-operators-deployable-rhel8 rhacm2/multicluster-operators-placementrule-rhel8 rhacm2/multicluster-operators-subscription-rhel8 rhacm2/multiclusterhub-repo-rhel8 rhacm2/multiclusterhub-rhel8 rhacm2/openshift-hive-rhel7 rhacm2/prometheus-alertmanager-rhel8 rhacm2/provider-credential-controller-rhel8 rhacm2/rcm-controller-rhel8 rhacm2/redisgraph-tls-rhel8 rhacm2/registration-rhel8 rhacm2/registration-rhel8-operator rhacm2/search-aggregator-rhel8 rhacm2/search-collector-rhel8 rhacm2/search-rhel8 rhacm2/search-ui-rhel8 rhacm2/submariner-addon-rhel8 rhacm2/thanos-receive-controller-rhel8 rhacm2/thanos-rhel7 rhacm2/work-rhel8 rhceph rhel7/couchbase-operator-admission rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/postgresql-96 rhel8/redis-5 rhmtc/openshift-migration-controller rhmtc/openshift-migration-operator rhmtc/openshift-migration-velero rhmtc/openshift-migration-velero-plugin-for-aws rhmtc/openshift-migration-velero-plugin-for-gcp rhmtc/openshift-migration-velero-plugin-for-microsoft-azure rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/postgresql-96-rhel7 rhscl/redis-32-rhel7 rook-ceph ubi8 ubi8/dotnet-50 ubi8/ruby-27 ubi8/ubi8-init volume-replication-operator clusters
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 10.212495 2.710259 3.191318 2.235464 2.531879 2.065175 13.434725 0.060849 11.409653 4.269774 12.530345 11.946793 12.017999 14.544186 3.135820 10.098921 1.368805 1.368805 9.527233 8.325504 8.707050 3.956360 0.202113 4.190339 4.715256 2.861501 2.830902 9.552491 2.746994 4.209699 2.903732 4.027416 11.390301 4.320844 9.714880 11.357460 6.231020 4.247392 4.910821 4.072643 4.862510 3.841428 9.624172 9.684011 4.680520 4.174164 3.257445 5.148783 4.443662 4.088297 9.162374 4.755046 4.590602 3.561153 3.470536 9.267571 4.615115 8.124936 12.312973 9.722846 0.949997 2.201105 11.877791 1.870983 20.328555 9.881997 6.373029 1.516743 12.167505 4.502316 4.500797 2.664554 13.502305 2.982820 24.147302 18.252098 18.225759 21.819908 17.388136 21.049847 19.747568 24.991442 26.460856 22.792500 20.962237 24.094806 19.344642 20.762423 20.828972 17.541626 20.673653 20.941541 16.162125 14.562713 15.601577 19.836530 23.907342 24.002155 18.002954 16.637362 15.204368 16.201372 17.623032 27.170011 16.125317 17.217897 16.146708 17.840188 17.818970 19.392183 19.123314 19.021766 25.768520 26.437218 21.199134 24.401813 20.889458 19.339222 20.428393 20.989450 17.962332 16.313270 20.773993 22.447410 22.478224 22.108274 27.209110 22.564356 22.702737 26.857052 22.704280 23.594188 24.224838 34.405157 36.674626 39.959749 37.161722 31.414786 34.949369 38.338329 35.988386 34.023225 45.426438 34.027710 52.143121 41.365593 41.890508 44.016748 44.476493 47.210869 47.127143 50.912804 40.859017 52.385038 51.598048 44.405758 49.149815 50.436538 45.433999 46.793926 48.762990 39.468954 48.409586 42.533801 46.674942 49.515474 51.988255 46.457628 43.261360 39.732950 40.997311 49.057928 49.582509 38.516327 39.299219 47.292226 50.356002 33.982659 47.033300 38.498074 41.573013 40.355792 36.747398 36.361805 36.836134 34.458896 38.214497 35.099470 37.119567 30.150124 44.292937 39.592091 39.553169 40.084230 49.942175 36.843029 37.329289 40.542117 44.955204 44.111268 48.574556 46.307341 44.369794 45.910794 39.508773 34.606631 32.985475 34.639291 44.090552 39.612524 34.450975 57.224090 42.415056 39.005122 45.200547 42.256659 34.474696 35.060935 42.951406 37.308406 33.256690 32.525959 46.347208 46.179090 40.287774 35.113987 44.399612 36.195427 33.293605 43.686889 34.489515 43.517309 49.908296 45.100237 41.908622 38.367799 37.725580 35.023128 35.831688 36.258679 42.448970 43.681448 46.449438 38.668214 41.365216 36.328266 42.905825 43.751595 47.199185 39.250797 38.670660 39.065841 49.291122 49.675727 51.102242 49.866126 46.976162 42.999086 32.381362 34.385506 32.244991 31.699549 42.522840 33.865061 45.268921 35.058863 37.889456 47.310233 46.371312 46.942884 34.292780 37.672536 40.728524 33.813742 36.538439 25.383468 9.017498 9.496428 9.417449 17.938269 2.651199 5.940052 4.996048 7.138859 5.071356 4.613383 4.920721 4.546185 10.497612 10.183270 10.914794 6.399054 3.819182 4.031038 3.837636 1.796607 2.415324 2.121465 4.816271 4.371615 6.509100 8.354674 11.373735 7.786041 4.605525 12.552122 11.856307 12.207861 12.183611 12.771721 11.615330 5.183471 5.511970 21.912805 5.992666 5.595747 4.927704 2.486133 4.520238 11.231596 4.546019 4.326051 2.908384 2.733491 3.324565 6.268206 3.047492 3.137155 0.157851 11.084354 0.592422 0.600581 1.880347 1.880347 1.880347 1.035297 21.391517 26.015025 20.513775 17.389355 17.245405 15.674871 1.186782 1.026305 2.146571 2.146571 1.391054 1.008353 0.445479 0.689480 0.368605 0.541914 11.297479 1
00067111-ddb4-437a-93e3-3d1c8ac25d31 8.995419 2.160370 2.573723 1.748092 1.971441 1.568602 12.580306 0.081880 10.507684 5.253336 11.764190 11.057840 11.092143 13.459416 2.860462 8.831634 0.877811 0.877811 8.263298 7.337406 7.638742 3.668963 0.207943 4.165411 4.638435 2.784338 1.969992 8.729956 1.954378 3.406200 2.760756 4.057920 10.359551 4.260565 8.435964 10.297434 5.419165 4.246583 4.759925 3.849967 4.536060 3.731698 9.019578 9.184391 4.763916 4.110895 3.401647 5.105924 4.356395 4.081403 8.632997 4.069583 4.490968 3.465728 3.387500 8.719292 4.217066 7.389243 11.368609 8.255353 0.585342 2.146530 10.855041 1.608688 19.252776 8.639306 5.462955 1.329632 11.053145 3.974135 3.793639 2.821655 12.273794 2.119603 22.787701 16.634177 16.576359 19.810713 15.893566 19.477255 18.353301 22.945757 23.940164 21.356926 18.579887 22.277995 18.080111 19.302375 19.325830 16.142802 18.431724 19.196351 14.847969 13.410344 14.365913 18.459042 21.330494 22.212030 16.697619 15.605483 14.115977 15.083211 16.437457 25.350972 14.787583 16.131670 15.091690 16.592196 16.571977 18.054700 17.758342 17.664975 23.604029 24.262968 19.611136 22.377212 19.253856 17.839518 18.786249 19.411484 16.760895 15.226945 19.170626 21.012882 20.873241 20.695305 25.087056 21.298379 21.298379 24.904029 21.273650 21.067244 21.935807 32.173666 34.928318 37.496854 36.119021 30.053017 33.293612 35.987900 35.029250 34.585738 44.749798 36.431493 49.582004 38.828579 39.567380 40.552166 41.258028 43.357612 44.820004 46.363902 37.804790 48.171164 47.643612 40.829130 45.174347 46.704688 42.081066 42.624045 44.128002 36.409009 44.564838 40.328463 43.376584 45.527159 47.821542 42.935165 40.096420 36.903753 38.316941 44.441854 45.636397 35.361700 36.675670 44.221972 46.672996 32.248996 44.981493 36.767400 39.213371 38.274533 35.079603 34.245386 35.281542 32.896698 36.083885 32.857434 35.332774 28.759449 42.252413 37.614756 36.709702 38.068139 47.757868 35.819541 35.411510 37.920821 42.233175 40.679242 45.296822 44.089484 43.130386 48.779994 41.743431 37.244004 35.522157 36.623853 42.145308 35.305521 33.781576 50.273683 39.496913 36.694416 42.984737 40.084402 33.585049 33.423991 40.506689 35.174353 31.451876 31.006745 45.301014 43.368625 38.157342 33.374982 42.309380 34.652461 31.892913 41.186123 32.930283 40.652884 43.534651 41.648645 39.881168 36.464573 35.255268 33.181994 33.649506 35.062219 40.149992 41.310526 43.144867 37.313902 39.181932 35.668313 45.288156 37.776153 42.165237 35.014425 33.459647 35.137877 44.161834 46.869177 47.852550 46.709097 44.083185 41.030145 30.876338 32.894593 30.265035 30.083894 39.989919 32.255956 42.763205 33.240120 35.470148 45.036492 43.068913 44.571341 31.793308 35.746188 38.118174 31.935027 34.194315 23.510481 8.318964 8.366031 8.032382 16.378978 1.442449 4.597851 3.666086 5.502300 3.540183 3.208001 3.293483 2.741118 8.586992 8.544193 9.039980 4.872741 2.944516 2.983574 2.806419 0.731866 1.235324 0.665272 3.494510 3.019229 5.037632 6.833176 9.321538 6.229780 2.869374 10.420578 9.692549 9.978749 9.972624 10.566197 9.440567 3.560109 3.725685 19.884362 3.619925 4.412654 3.385024 1.050930 2.029219 8.465909 3.116701 2.998431 1.417360 1.376821 2.241953 4.794953 1.508559 0.978321 0.033841 9.998611 0.078088 0.104292 1.363427 1.363427 1.363427 0.420853 20.195932 24.297575 19.305067 16.271568 16.099687 14.695559 0.688198 0.482011 1.583662 1.583662 0.755498 0.865163 0.383049 0.869834 0.368129 0.808570 10.370895 1
002663ad-bcf4-4c7c-9530-ecb351fe4001 0.234350 0.054241 0.061423 0.042898 0.056661 0.046651 0.320669 0.000000 0.230667 0.042898 0.210621 0.232727 0.254084 0.286222 0.101032 0.195826 0.030077 0.030077 0.217672 0.167647 0.158732 0.076878 0.000000 0.131354 0.069532 0.073358 0.030154 0.186912 0.030549 0.029199 0.044547 0.085325 0.153505 0.037568 0.146624 0.153505 0.045733 0.034117 0.037187 0.031276 0.058078 0.024501 0.159012 0.172721 0.066841 0.037073 0.031015 0.055430 0.042787 0.051985 0.162318 0.038389 0.040735 0.042285 0.050316 0.166624 0.032285 0.132413 0.260545 0.195989 0.015446 0.021565 0.241863 0.026759 0.281851 0.190426 0.069712 0.015096 0.265217 0.051866 0.055640 0.083305 0.301221 0.049249 0.296759 0.246576 0.241439 0.349771 0.232268 0.294837 0.279087 0.384531 0.413846 0.407220 0.276864 0.438163 0.298218 0.307715 0.307715 0.195443 0.285582 0.326840 0.184702 0.190683 0.193719 0.229789 0.336268 0.390845 0.213756 0.234497 0.209352 0.221808 0.229036 0.435769 0.182712 0.208265 0.236433 0.229964 0.239159 0.251761 0.220867 0.248288 0.408600 0.420555 0.262308 0.391023 0.291916 0.277669 0.292920 0.241622 0.238789 0.216620 0.269894 0.312689 0.307712 0.304105 0.451813 0.317830 0.317830 0.446130 0.317830 0.282132 0.295533 0.590296 0.513729 0.619681 0.463670 0.422792 0.578947 0.670088 0.448633 0.458749 0.780148 0.495614 0.642073 0.514902 0.697794 0.601349 0.584889 0.682043 0.685380 0.674267 0.525038 0.721489 0.700118 0.685874 0.662047 0.607593 0.566363 0.594640 0.615306 0.682507 0.710667 0.490223 0.653217 0.670996 0.713805 0.631984 0.549523 0.517423 0.461941 0.678058 1.024813 0.540270 0.529198 0.584343 0.796774 0.421619 0.603851 0.507706 0.637253 0.529392 0.531015 0.504174 0.532917 0.539758 0.492350 0.502762 0.477512 0.408955 0.659091 0.597568 0.585768 0.492350 0.626597 0.463670 0.523331 0.630467 0.603851 0.546033 0.596825 0.739541 0.644425 0.597780 0.574783 0.501428 0.478933 0.554271 0.584343 0.471574 0.516099 0.511229 0.537195 0.793437 0.559204 0.525297 0.473389 1.610294 1.610294 1.314442 1.036811 0.536484 0.721489 0.681475 0.554418 0.440920 0.547429 0.455515 0.415610 0.607842 0.457610 0.560314 0.554039 0.604380 0.567612 0.526552 0.551601 0.507822 0.563376 0.485654 0.715503 0.694368 0.747661 0.549571 0.650263 0.462070 0.538065 0.517396 0.547429 0.527192 0.454530 0.483317 0.710227 0.642073 0.824695 0.664003 0.764381 0.577685 0.446154 0.466508 0.434539 0.447353 0.696360 0.472658 0.605818 0.520000 0.604116 0.632500 0.597780 0.632500 0.596936 0.529198 0.634570 0.482165 0.525555 0.421226 0.156667 0.150955 0.177790 0.256597 0.015931 0.042232 0.009368 0.064977 0.009112 0.022286 0.032002 0.023883 0.198093 0.168402 0.196865 0.076086 0.035478 0.047868 0.038013 0.013290 0.000000 0.000000 0.009502 0.020889 0.017583 0.049691 0.151667 0.054605 0.055813 0.160126 0.143092 0.150955 0.158741 0.148431 0.141253 0.019371 0.020889 0.274649 0.051866 0.042828 0.010555 0.026596 0.038740 0.195760 0.067615 0.041561 0.012086 0.011462 0.015359 0.027212 0.012086 0.012540 0.000000 0.201006 0.000000 0.013290 0.036700 0.036700 0.036700 0.020408 0.277572 0.439695 0.277215 0.231754 0.231754 0.208186 0.000000 0.011870 0.033060 0.033060 0.017857 0.000000 0.000000 0.000000 0.000000 0.000000 0.220283 0
00351e6e-53ce-465e-9493-cf0cd2367049 10.212495 2.710259 3.191318 2.235464 2.531879 2.065175 13.434725 0.060849 11.409653 4.269774 12.530345 11.946793 12.017999 14.544186 3.135820 10.098921 1.368805 1.368805 9.527233 8.325504 8.707050 3.956360 0.202113 4.190339 4.715256 2.861501 2.830902 9.552491 2.746994 4.209699 2.903732 4.027416 11.390301 4.320844 9.714880 11.357460 6.231020 4.247392 4.910821 4.072643 4.862510 3.841428 9.624172 9.684011 4.680520 4.174164 3.257445 5.148783 4.443662 4.088297 9.162374 4.755046 4.590602 3.561153 3.470536 9.267571 4.615115 8.124936 12.312973 9.722846 0.949997 2.201105 11.877791 1.870983 20.328555 9.881997 6.373029 1.516743 12.167505 4.502316 4.500797 2.664554 13.502305 2.982820 24.147302 18.252098 18.225759 21.819908 17.388136 21.049847 19.747568 24.991442 26.460856 22.792500 20.962237 24.094806 19.344642 20.762423 20.828972 17.541626 20.673653 20.941541 16.162125 14.562713 15.601577 19.836530 23.907342 24.002155 18.002954 16.637362 15.204368 16.201372 17.623032 27.170011 16.125317 17.217897 16.146708 17.840188 17.818970 19.392183 19.123314 19.021766 25.768520 26.437218 21.199134 24.401813 20.889458 19.339222 20.428393 20.989450 17.962332 16.313270 20.773993 22.447410 22.478224 22.108274 27.209110 22.564356 22.702737 26.857052 22.704280 23.594188 24.224838 34.405157 36.674626 39.959749 37.161722 31.414786 34.949369 38.338329 35.988386 34.023225 45.426438 34.027710 52.143121 41.365593 41.890508 44.016748 44.476493 47.210869 47.127143 50.912804 40.859017 52.385038 51.598048 44.405758 49.149815 50.436538 45.433999 46.793926 48.762990 39.468954 48.409586 42.533801 46.674942 49.515474 51.988255 46.457628 43.261360 39.732950 40.997311 49.057928 49.582509 38.516327 39.299219 47.292226 50.356002 33.982659 47.033300 38.498074 41.573013 40.355792 36.747398 36.361805 36.836134 34.458896 38.214497 35.099470 37.119567 30.150124 44.292937 39.592091 39.553169 40.084230 49.942175 36.843029 37.329289 40.542117 44.955204 44.111268 48.574556 46.307341 44.369794 45.910794 39.508773 34.606631 32.985475 34.639291 44.090552 39.612524 34.450975 57.224090 42.415056 39.005122 45.200547 42.256659 34.474696 35.060935 42.951406 37.308406 33.256690 32.525959 46.347208 46.179090 40.287774 35.113987 44.399612 36.195427 33.293605 43.686889 34.489515 43.517309 49.908296 45.100237 41.908622 38.367799 37.725580 35.023128 35.831688 36.258679 42.448970 43.681448 46.449438 38.668214 41.365216 36.328266 42.905825 43.751595 47.199185 39.250797 38.670660 39.065841 49.291122 49.675727 51.102242 49.866126 46.976162 42.999086 32.381362 34.385506 32.244991 31.699549 42.522840 33.865061 45.268921 35.058863 37.889456 47.310233 46.371312 46.942884 34.292780 37.672536 40.728524 33.813742 36.538439 25.383468 9.017498 9.496428 9.417449 17.938269 2.651199 5.940052 4.996048 7.138859 5.071356 4.613383 4.920721 4.546185 10.497612 10.183270 10.914794 6.399054 3.819182 4.031038 3.837636 1.796607 2.415324 2.121465 4.816271 4.371615 6.509100 8.354674 11.373735 7.786041 4.605525 12.552122 11.856307 12.207861 12.183611 12.771721 11.615330 5.183471 5.511970 21.912805 5.992666 5.595747 4.927704 2.486133 4.520238 11.231596 4.546019 4.326051 2.908384 2.733491 3.324565 6.268206 3.047492 3.137155 0.157851 11.084354 0.592422 0.600581 1.880347 1.880347 1.880347 1.035297 21.391517 26.015025 20.513775 17.389355 17.245405 15.674871 1.186782 1.026305 2.146571 2.146571 1.391054 1.008353 0.445479 0.689480 0.368605 0.541914 11.297479 1
003ba133-e754-4d5a-bc57-675b386d1f05 10.951183 3.517931 3.740535 2.764705 3.133221 2.630070 13.867011 0.074160 11.707567 4.515827 13.604715 12.251898 12.340407 15.047085 3.314593 10.975715 1.810429 1.810429 10.264170 9.049425 9.321219 4.135661 0.228689 4.404162 5.666622 2.978878 3.613465 9.961682 3.475348 5.205738 3.398058 4.405590 12.276364 4.802197 10.619164 12.072765 7.167730 4.685000 5.896319 4.481388 5.395678 4.220439 10.185902 10.232341 5.134309 4.642546 3.672282 5.561682 5.326936 4.535108 9.727619 5.142252 5.523034 4.045146 3.905986 9.836355 6.088212 9.173560 12.815071 10.692749 0.976592 2.355708 12.274347 1.993156 20.504672 10.643172 6.634164 1.645692 12.564222 5.012691 4.914112 2.793988 13.907210 3.106547 24.666830 18.948801 18.917567 22.563003 18.127229 21.791420 20.544236 25.679524 27.554210 23.393518 21.829677 25.029697 20.188353 21.559944 21.635540 19.006940 21.501535 21.673315 17.536614 15.192013 16.202822 21.504940 24.818715 24.809079 19.546032 17.491144 15.910928 16.878435 18.269513 27.892956 17.526460 18.767355 17.091931 18.709167 18.810325 20.258861 20.783715 19.858484 26.708778 27.415513 22.670968 25.324686 21.937058 20.284681 21.514110 22.805650 18.783911 17.055797 21.668875 23.412183 23.332240 23.069336 28.165647 23.199662 23.572612 27.776278 23.554172 24.471570 25.082704 34.999842 37.068444 40.653670 38.267905 31.762651 35.321064 38.997088 37.063366 35.065237 46.057781 34.380187 52.408816 41.112126 42.478394 45.057817 45.258612 48.193247 48.041676 52.029166 42.315533 53.341091 52.582388 45.313231 50.084954 51.302536 46.876742 47.742788 49.787969 40.332506 49.348652 43.309763 47.633845 50.639522 52.979576 47.352215 44.090043 41.085790 42.753519 50.035337 50.534582 39.662977 39.914909 47.407350 50.998583 34.684692 47.372633 38.970027 42.270643 40.821859 37.245321 36.885897 37.286145 34.886896 38.552029 35.662371 38.208289 30.552324 44.649399 40.040376 40.123161 40.782520 50.280675 37.932309 37.742899 41.255060 45.276380 44.612008 48.932161 46.929996 44.929260 46.326396 40.018337 35.044324 33.439736 35.116443 44.368703 40.129647 34.817999 57.623213 42.894928 39.589656 45.779836 43.442380 35.558426 35.480820 43.595959 37.918530 33.801034 32.965421 46.975573 46.998235 40.674312 36.312921 44.984013 36.852530 33.841841 44.183583 34.969849 43.904332 50.260364 45.534900 42.334941 39.538761 38.200550 35.546339 36.479266 37.335765 43.097369 44.329016 47.209978 39.109179 42.098739 37.440905 43.372133 44.129824 47.623027 39.762595 39.376851 39.549285 50.019401 50.014459 51.795133 50.313201 47.741496 44.458119 33.902644 35.937505 33.335965 32.635994 43.995921 34.841518 45.766723 35.516435 38.585127 47.623246 46.716512 47.282906 35.411800 38.110054 41.442377 34.359862 37.070359 26.527273 9.672069 10.774288 10.534953 18.773565 4.293197 8.756880 8.464677 8.965426 8.703528 6.355292 6.973385 6.391876 12.163150 11.749437 12.553777 8.576116 6.221683 6.194553 5.877374 2.715820 3.963806 3.757729 8.498035 5.918440 11.066506 13.482184 14.596288 10.617950 6.397407 14.376132 14.049615 14.339744 14.231061 14.857704 14.110938 7.068935 7.531914 23.223087 7.734944 8.075715 8.152159 3.923503 6.274726 12.917456 6.465550 7.028419 4.865130 4.573675 5.388398 9.007374 4.337316 4.343723 0.318409 11.972073 1.114720 1.121995 2.411446 2.411446 2.411446 1.565021 22.602342 26.656079 21.071705 17.942118 17.793542 16.191431 1.735951 1.551169 2.678562 2.678562 1.915918 1.144896 0.504067 0.806203 0.394792 0.592381 11.895507 1

Visualizing the results of the 3D PCA

pca_c = pca3.fit_transform(cluster_cont_df)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pca_c[:, 0], ys=pca_c[:, 1], zs=pca_c[:, 2], c=cluster_cont_df["clusters"]
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.show()
../_images/ML_clustering_154_0.png

3D interactive visualization for image dataset

fig = px.scatter_3d(
    pca_c,
    x=pca_c[:, 0],
    y=pca_c[:, 1],
    z=pca_c[:, 2],
    color=cluster_cont_df["clusters"],
    opacity=0.1,
)
fig.show()

Listing out the groups of image_name in different clusters

df1 = pd.DataFrame([])
global_mean = cluster_cont_df.mean().sort_values(ascending=False)


for c in tqdm(range(0, 3)):
    c = cluster_cont_df[cluster_cont_df.clusters == c].iloc[:, :-1].mean()
    diff = c - global_mean
    d = diff.sort_values(ascending=False).head(20)
    d1 = pd.DataFrame(d).reset_index()
    d1.rename(columns={"index": "image_name"}, inplace=True)
    df1 = df1.append(d1.image_name)
df1 = df1.reset_index()
df1.rename(index={0: "cluster 1", 1: "cluster 2", 2: "cluster 3"}, inplace=True)
df1.drop(columns="index", inplace=True)

df1
100%|██████████| 3/3 [00:00<00:00, 141.29it/s]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
cluster 1 Cilium ubi8/ruby-27 cephcsi ubi8 rhel8/mysql-80 rhel8/httpd-24 ubi8/ubi8-init ubi8/dotnet-50 rhceph rhel8/redis-5 rhscl/mysql-57-rhel7 grafana rhscl/mongodb-36-rhel7 rook-ceph rhscl/redis-32-rhel7 amq7/amq-streams-kafka-26-rhel7 amq7/amq-streams-kafka-25-rhel7 mcg-core rhacm2/grafana rhacm2/grc-ui-rhel8
cluster 2 rhacm2/registration-rhel8 rhacm2/work-rhel8 rhacm2/multicloud-manager-rhel8 rhacm2/multiclusterhub-rhel8 rhacm2/registration-rhel8-operator rhacm2/grc-ui-rhel8 rhacm2/redisgraph-tls-rhel8 rhacm2/multiclusterhub-repo-rhel8 rhacm2/application-ui-rhel8 rhacm2/search-rhel8 rhacm2/grafana rhacm2/thanos-rhel7 rhacm2/grc-ui-api-rhel8 rhacm2/search-ui-rhel8 rhacm2/rcm-controller-rhel8 rhacm2/console-rhel8 rhacm2/search-aggregator-rhel8 rhacm2/submariner-addon-rhel8 rhacm2/prometheus-alertmanager-rhel8 rhacm2/managedcluster-import-controller-rhel8
cluster 3 openshift/ose-kube-rbac-proxy openshift/ose-cli openshift/ose-ironic openshift/ose-ptp openshift/ose-sdn openshift/ose-etcd openshift/ose-coredns openshift/ose-tests openshift/ose-oauth-proxy openshift/ose-tools openshift/ose-console openshift/ose-machine-api-operator openshift/ose-hyperkube openshift/ose-ptp-operator openshift/ose-installer openshift/ose-thanos openshift/ose-kuryr-cni openshift/ose-telemeter openshift/ose-deployer openshift/ose-prometheus

We generated the above dataframe by differencitating the global mean with the mean of the individual cluster, represented as a index in the dataframe. Each row in the dataframe df1, represents groups of top 20 image_names having a higher probability of occuring/running for the same user.


KMeans clustering by similarity encoders: (Not Taking frequency into considerations)

Encoding image layer dataset

In this issue, we are applying the encoding for the image_name from the dataset. Similar to previous approach, we encoded the image_name with respect to the cluster_id.

df_image.drop_duplicates(inplace=True)
df_image_name = df_image[["name"]].reindex()
enc = SimilarityEncoder(similarity="ngram")
X_enc = enc.fit_transform(df_image_name)
X_image_name = pd.DataFrame(X_enc)
X_image_name.columns = sorted(df_image_name.name.unique())
X_image = pd.concat([df_image["cluster_id"], X_image_name], axis=1)
X_image = X_image.set_index("cluster_id")
X_image = X_image.groupby(level=0).sum()
X_image.head()
3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 jboss-amq-6/amq63-openshift openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-grafana openshift/ose-jenkins openshift/ose-must-gather openshift/ose-oauth-proxy openshift/ose-tests openshift/ose-tools rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/redis-5 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/redis-32-rhel7 ubi8/dotnet-50 ubi8/ruby-27
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 0.011236 0.026316 0.018868 0.013245 0.013793 0.165468 0.571429 1.0 0.384615 0.384615 0.333333 0.333333 0.4375 0.408163 0.0 0.0 0.007299 0.007299 0.0 0.006849 0.007143 0.012987 0.007143 0.0 0.0
00351e6e-53ce-465e-9493-cf0cd2367049 0.011236 0.026316 0.018868 0.013245 0.013793 0.165468 0.571429 1.0 0.384615 0.384615 0.333333 0.333333 0.4375 0.408163 0.0 0.0 0.007299 0.007299 0.0 0.006849 0.007143 0.012987 0.007143 0.0 0.0
003ba133-e754-4d5a-bc57-675b386d1f05 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000 0.0000 0.000000 0.0 0.0 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000 0.0 0.0
00479ead-b7fc-49c2-ae20-3990a9b3d08c 0.011236 0.026316 0.018868 0.013245 0.013793 0.165468 0.571429 1.0 0.384615 0.384615 0.333333 0.333333 0.4375 0.408163 0.0 0.0 0.007299 0.007299 0.0 0.006849 0.007143 0.012987 0.007143 0.0 0.0
00748c32-15c3-4586-98fb-e5078ca9f3b8 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000 0.0000 0.000000 0.0 0.0 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000 0.0 0.0
X_image.shape
(1810, 25)

1810 rows corresponds to 1810 cluster_id.

Encoding container dataset

df_cont.drop_duplicates(inplace=True)
df_cont_name = df_cont[["name"]].reindex()
enc = SimilarityEncoder(similarity="ngram")
X_enc = enc.fit_transform(df_cont_name)
X_cont_name = pd.DataFrame(X_enc)
X_cont_name.columns = sorted(df_cont_name.name.unique())
X_cont = pd.concat([df_cont["cluster_id"], X_cont_name], axis=1)
X_cont = X_cont.set_index("cluster_id")
X_cont = X_cont.groupby(level=0).sum()
X_cont.head()
3scale-amp2/3scale-rhel7-operator 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 Calico Operator Cilium Cloud Native PostgreSQL Operator Elastic Cloud on Kubernetes F5 BIG-IP Controller Operator NGINX Ingress Operator NVIDIA GPU Operator Seldon Operator alertmanager amq7/amq-broker-rhel7-operator amq7/amq-streams-kafka-25-rhel7 amq7/amq-streams-kafka-26-rhel7 amq7/amq-streams-rhel7-operator amq7/amq-streams-rhel7-operator-metadata ansible-automation-platform/platform-resource-rhel7-operator ansible-tower cephcsi cert-manager cainjector cert-manager controller cert-manager webhook codeready-workspaces/devfileregistry-rhel8 codeready-workspaces/operator codeready-workspaces/pluginregistry-rhel8 codeready-workspaces/server-rhel8 collectd-exporter container-native-virtualization/bridge-marker container-native-virtualization/cluster-network-addons-operator container-native-virtualization/cnv-containernetworking-plugins container-native-virtualization/hostpath-provisioner-rhel8-operator container-native-virtualization/hyperconverged-cluster-operator container-native-virtualization/hyperconverged-cluster-webhook-rhel8 container-native-virtualization/kubemacpool container-native-virtualization/kubernetes-nmstate-handler-rhel8 container-native-virtualization/kubevirt-cpu-model-nfd-plugin container-native-virtualization/kubevirt-cpu-node-labeller container-native-virtualization/kubevirt-kvm-info-nfd-plugin container-native-virtualization/kubevirt-ssp-operator container-native-virtualization/node-maintenance-operator container-native-virtualization/ovs-cni-marker container-native-virtualization/ovs-cni-plugin container-native-virtualization/virt-api container-native-virtualization/virt-cdi-apiserver container-native-virtualization/virt-cdi-controller container-native-virtualization/virt-cdi-importer container-native-virtualization/virt-cdi-operator container-native-virtualization/virt-cdi-uploadproxy container-native-virtualization/virt-controller container-native-virtualization/virt-handler container-native-virtualization/virt-launcher container-native-virtualization/virt-operator container-native-virtualization/vm-import-controller-rhel8 container-native-virtualization/vm-import-operator-rhel8 costmanagement-metrics-operator distributed-tracing/jaeger-rhel8-operator grafana ibm common service webhook ibm-events-operator ibm-postgresql jboss-amq-6/amq63-openshift jboss-eap-7/eap73-rhel8-operator kube-state-metrics mcg-core mcg-operator mcr.microsoft.com/mssql/rhel8/server must-gather-service ocp-tools-4/odo-init-image ocs-operator ocs-registry openshift-compliance-content openshift-gitops-1-tech-preview/argocd-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8-operator openshift-gitops-1-tech-preview/kam-delivery-rhel8 openshift-gitops-1/gitops-rhel8 openshift-gitops-1/kam-delivery-rhel8 openshift-logging/cluster-logging-operator-bundle openshift-logging/cluster-logging-rhel8-operator openshift-logging/elasticsearch-operator-bundle openshift-logging/elasticsearch-proxy-rhel8 openshift-logging/elasticsearch-rhel8-operator openshift-logging/elasticsearch6-rhel8 openshift-logging/fluentd-rhel8 openshift-logging/kibana6-rhel8 openshift-pipelines-tech-preview/pipelines-controller-rhel8 openshift-pipelines-tech-preview/pipelines-operator-proxy-rhel8 openshift-pipelines-tech-preview/pipelines-rhel8-operator openshift-pipelines-tech-preview/pipelines-triggers-controller-rhel8 openshift-pipelines-tech-preview/pipelines-triggers-webhook-rhel8 openshift-pipelines-tech-preview/pipelines-webhook-rhel8 openshift-pipelines/pipelines-controller-rhel8 openshift-pipelines/pipelines-operator-proxy-rhel8 openshift-pipelines/pipelines-rhel8-operator openshift-pipelines/pipelines-triggers-controller-rhel8 openshift-pipelines/pipelines-triggers-core-interceptors-rhel8 openshift-pipelines/pipelines-triggers-eventlistenersink-rhel8 openshift-pipelines/pipelines-triggers-webhook-rhel8 openshift-pipelines/pipelines-webhook-rhel8 openshift-sandboxed-containers-operator openshift-serverless-1-tech-preview/eventing-kafka-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-dispatcher-rhel8 openshift-serverless-1/eventing-mtbroker-ingress-rhel8 openshift-serverless-1/eventing-mtchannel-broker-rhel8 openshift-serverless-1/eventing-mtping-rhel8 openshift-serverless-1/eventing-sugar-controller-rhel8 openshift-serverless-1/eventing-webhook-rhel8 openshift-serverless-1/ingress-rhel8-operator openshift-serverless-1/knative-rhel8-operator openshift-serverless-1/kourier-control-rhel8 openshift-serverless-1/serverless-rhel8-operator openshift-serverless-1/serving-activator-rhel8 openshift-serverless-1/serving-autoscaler-hpa-rhel8 openshift-serverless-1/serving-autoscaler-rhel8 openshift-serverless-1/serving-controller-rhel8 openshift-serverless-1/serving-domain-mapping-rhel8 openshift-serverless-1/serving-domain-mapping-webhook-rhel8 openshift-serverless-1/serving-queue-rhel8 openshift-service-mesh/galley-rhel8 openshift-service-mesh/grafana-rhel8 openshift-service-mesh/istio-cni-rhel8 openshift-service-mesh/istio-rhel8-operator openshift-service-mesh/kiali-rhel7 openshift-service-mesh/kiali-rhel8 openshift-service-mesh/kiali-rhel8-operator openshift-service-mesh/pilot-rhel8 openshift-service-mesh/prometheus-rhel8 openshift-service-mesh/proxyv2-rhel8 openshift/compliance-operator openshift/ose-aws-ebs-csi-driver openshift/ose-aws-ebs-csi-driver-operator openshift/ose-aws-machine-controllers openshift/ose-aws-pod-identity-webhook openshift/ose-azure-disk-csi-driver openshift/ose-azure-disk-csi-driver-operator openshift/ose-azure-machine-controllers openshift/ose-baremetal-machine-controllers openshift/ose-baremetal-operator openshift/ose-baremetal-runtimecfg openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-cloud-credential-operator openshift/ose-cluster-authentication-operator openshift/ose-cluster-autoscaler openshift/ose-cluster-autoscaler-operator openshift/ose-cluster-baremetal-operator openshift/ose-cluster-config-operator openshift/ose-cluster-csi-snapshot-controller-operator openshift/ose-cluster-dns-operator openshift/ose-cluster-etcd-operator openshift/ose-cluster-image-registry-operator openshift/ose-cluster-ingress-operator openshift/ose-cluster-kube-apiserver-operator openshift/ose-cluster-kube-controller-manager-operator openshift/ose-cluster-kube-descheduler-operator openshift/ose-cluster-kube-scheduler-operator openshift/ose-cluster-kube-storage-version-migrator-operator openshift/ose-cluster-logging-operator openshift/ose-cluster-machine-approver openshift/ose-cluster-monitoring-operator openshift/ose-cluster-network-operator openshift/ose-cluster-nfd-operator openshift/ose-cluster-node-tuning-operator openshift/ose-cluster-openshift-apiserver-operator openshift/ose-cluster-openshift-controller-manager-operator openshift/ose-cluster-policy-controller openshift/ose-cluster-samples-operator openshift/ose-cluster-storage-operator openshift/ose-clusterresourceoverride-rhel8-operator openshift/ose-configmap-reloader openshift/ose-console openshift/ose-console-operator openshift/ose-container-networking-plugins openshift/ose-coredns openshift/ose-csi-driver-manila openshift/ose-csi-driver-manila-operator openshift/ose-csi-driver-nfs openshift/ose-csi-external-attacher openshift/ose-csi-external-provisioner openshift/ose-csi-external-resizer openshift/ose-csi-external-snapshotter openshift/ose-csi-livenessprobe openshift/ose-csi-node-driver-registrar openshift/ose-csi-snapshot-controller openshift/ose-csi-snapshot-validation-webhook openshift/ose-deployer openshift/ose-docker-builder openshift/ose-docker-registry openshift/ose-egress-router-cni openshift/ose-etcd openshift/ose-gcp-machine-controllers openshift/ose-gcp-pd-csi-driver openshift/ose-gcp-pd-csi-driver-operator openshift/ose-grafana openshift/ose-haproxy-router openshift/ose-hyperkube openshift/ose-insights-operator openshift/ose-installer openshift/ose-ironic openshift/ose-ironic-inspector openshift/ose-ironic-ipa-downloader openshift/ose-ironic-machine-os-downloader openshift/ose-ironic-static-ip-manager openshift/ose-jenkins openshift/ose-k8s-prometheus-adapter openshift/ose-keepalived-ipfailover openshift/ose-kube-rbac-proxy openshift/ose-kube-state-metrics openshift/ose-kube-storage-version-migrator openshift/ose-kuryr-cni openshift/ose-kuryr-controller openshift/ose-libvirt-machine-controllers openshift/ose-local-storage-diskmaker openshift/ose-local-storage-operator openshift/ose-local-storage-operator-bundle openshift/ose-local-storage-static-provisioner openshift/ose-logging-elasticsearch6 openshift/ose-machine-api-operator openshift/ose-machine-config-operator openshift/ose-mdns-publisher openshift/ose-multus-admission-controller openshift/ose-multus-cni openshift/ose-multus-route-override-cni openshift/ose-multus-whereabouts-ipam-cni openshift/ose-must-gather openshift/ose-network-metrics-daemon openshift/ose-oauth-apiserver openshift/ose-oauth-proxy openshift/ose-oauth-server openshift/ose-openshift-apiserver openshift/ose-openshift-controller-manager openshift/ose-openshift-state-metrics openshift/ose-openstack-cinder-csi-driver openshift/ose-openstack-cinder-csi-driver-operator openshift/ose-openstack-machine-controllers openshift/ose-operator-lifecycle-manager openshift/ose-operator-marketplace openshift/ose-operator-registry openshift/ose-ovirt-csi-driver openshift/ose-ovirt-csi-driver-operator openshift/ose-ovirt-machine-controllers openshift/ose-ovn-kubernetes openshift/ose-prom-label-proxy openshift/ose-prometheus openshift/ose-prometheus-alertmanager openshift/ose-prometheus-config-reloader openshift/ose-prometheus-node-exporter openshift/ose-prometheus-operator openshift/ose-ptp openshift/ose-ptp-operator openshift/ose-sdn openshift/ose-service-ca-operator openshift/ose-sriov-cni openshift/ose-sriov-dp-admission-controller openshift/ose-sriov-infiniband-cni openshift/ose-sriov-network-config-daemon openshift/ose-sriov-network-device-plugin openshift/ose-sriov-network-operator openshift/ose-sriov-network-webhook openshift/ose-telemeter openshift/ose-template-service-broker openshift/ose-template-service-broker-operator openshift/ose-tests openshift/ose-thanos openshift/ose-tools openshift/ose-vertical-pod-autoscaler-rhel8-operator openshift/ose-vsphere-csi-driver openshift/ose-vsphere-csi-driver-operator openshift/ose-vsphere-csi-driver-syncer openshift/ose-vsphere-problem-detector openshift4/performance-addon-rhel8-operator quay/quay-container-security-operator-container quay/quay-container-security-operator-rhel8 quay/quay-operator-rhel8 rh-sso-7/sso74-openj9-openshift-rhel8 rhacm2/application-ui-rhel8 rhacm2/cert-manager-controller-rhel8 rhacm2/cert-policy-controller-rhel8 rhacm2/clusterlifecycle-state-metrics-rhel8 rhacm2/config-policy-controller-rhel8 rhacm2/console-api-rhel8 rhacm2/console-header-rhel8 rhacm2/console-rhel8 rhacm2/discovery-rhel8-operator rhacm2/endpoint-component-rhel8-operator rhacm2/endpoint-rhel8-operator rhacm2/governance-policy-propagator-rhel8 rhacm2/governance-policy-spec-sync-rhel8 rhacm2/governance-policy-status-sync-rhel8 rhacm2/governance-policy-template-sync-rhel8 rhacm2/grafana rhacm2/grc-ui-api-rhel8 rhacm2/grc-ui-rhel8 rhacm2/iam-policy-controller-rhel8 rhacm2/insights-client-rhel8 rhacm2/klusterlet-addon-controller-rhel8 rhacm2/klusterlet-addon-lease-controller-rhel8 rhacm2/klusterlet-addon-operator-rhel8 rhacm2/managedcluster-import-controller-rhel8 rhacm2/multicloud-manager-rhel8 rhacm2/multicluster-observability-rhel8-operator rhacm2/multicluster-operators-application-rhel8 rhacm2/multicluster-operators-channel-rhel8 rhacm2/multicluster-operators-deployable-rhel8 rhacm2/multicluster-operators-placementrule-rhel8 rhacm2/multicluster-operators-subscription-rhel8 rhacm2/multiclusterhub-repo-rhel8 rhacm2/multiclusterhub-rhel8 rhacm2/openshift-hive-rhel7 rhacm2/prometheus-alertmanager-rhel8 rhacm2/provider-credential-controller-rhel8 rhacm2/rcm-controller-rhel8 rhacm2/redisgraph-tls-rhel8 rhacm2/registration-rhel8 rhacm2/registration-rhel8-operator rhacm2/search-aggregator-rhel8 rhacm2/search-collector-rhel8 rhacm2/search-rhel8 rhacm2/search-ui-rhel8 rhacm2/submariner-addon-rhel8 rhacm2/thanos-receive-controller-rhel8 rhacm2/thanos-rhel7 rhacm2/work-rhel8 rhceph rhel7/couchbase-operator-admission rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/postgresql-96 rhel8/redis-5 rhmtc/openshift-migration-controller rhmtc/openshift-migration-operator rhmtc/openshift-migration-velero rhmtc/openshift-migration-velero-plugin-for-aws rhmtc/openshift-migration-velero-plugin-for-gcp rhmtc/openshift-migration-velero-plugin-for-microsoft-azure rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/postgresql-96-rhel7 rhscl/redis-32-rhel7 rook-ceph ubi8 ubi8/dotnet-50 ubi8/ruby-27 ubi8/ubi8-init volume-replication-operator
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 3.473566 1.018976 0.969853 0.727917 0.903659 0.638310 4.856031 0.071881 4.188458 1.166577 4.536447 4.372772 4.311122 5.218377 1.507491 3.529069 0.418113 0.418113 3.362787 2.959808 2.951677 1.535231 0.084835 1.759574 1.976798 1.251000 1.087402 3.384589 1.038971 1.477857 1.155036 1.479620 4.273381 1.547780 3.239151 4.221179 2.327051 1.166660 1.444477 1.187561 1.527275 1.034186 3.125478 3.437599 1.751956 1.471204 1.318664 2.135316 1.754705 1.617287 3.360968 1.356369 1.805850 1.414359 1.367001 3.363800 1.695205 2.885589 4.675337 3.520072 0.827616 0.754337 4.305410 0.662694 7.294208 3.439188 1.686792 0.717626 4.297657 1.698541 1.650525 1.028480 4.872312 1.296509 8.898060 6.357425 6.342727 7.637750 6.115266 7.50902 7.110615 9.204171 9.682815 8.216430 6.883065 8.603263 6.935158 7.447922 7.437892 6.200971 6.995870 7.357153 5.738256 5.074301 5.407914 7.098950 8.057905 8.518778 6.467322 6.040616 5.493819 5.717405 6.191609 9.754582 5.749054 6.321067 5.950637 6.461236 6.453320 7.024666 6.946461 6.842951 9.264394 9.438413 7.659268 8.732760 7.548596 6.910231 7.319650 7.604566 6.493999 5.857753 7.422695 7.974581 8.194380 7.851328 9.677626 8.063801 8.102822 9.569785 8.025948 8.156649 8.053841 12.313921 13.368781 14.404820 13.327752 11.197589 12.682753 13.768360 12.904522 12.083345 16.120155 12.054865 19.897757 16.385741 15.657425 16.041510 16.590550 17.195425 17.295259 18.473376 14.814216 19.707362 19.195414 16.653385 18.381528 17.203721 15.584748 15.660611 16.189260 13.123054 17.975729 15.906740 17.457574 18.264955 19.273006 17.110577 17.364096 15.720253 15.562756 18.243406 18.043845 13.952729 14.155009 17.532776 18.082900 12.148041 18.052603 14.162405 15.144608 14.807066 14.063096 13.187151 13.774541 12.759996 13.640274 12.913438 13.349918 10.635212 16.032908 14.450046 14.635044 14.048417 17.690087 13.205479 13.563666 14.575548 17.112545 15.289298 15.663069 16.906743 16.235783 16.419568 14.100545 12.288698 11.616446 12.519215 16.026566 13.945681 12.306838 14.546837 13.930994 12.404988 15.503182 14.777960 12.376067 12.559782 15.140393 13.139013 11.371097 11.698703 16.606040 15.908692 14.490854 12.818688 15.374940 12.291182 11.213016 15.746825 12.742373 15.890944 15.745736 16.284631 16.132168 14.524984 13.947269 12.813362 13.131946 13.046415 15.817210 15.330541 16.781567 14.045752 14.912370 12.946702 14.265939 14.699496 16.227866 14.133932 13.229957 13.215833 17.027308 17.799789 18.172427 18.103408 16.496365 15.320873 11.863500 12.136184 11.469051 11.112011 15.114341 12.011578 16.237969 12.415719 13.294117 17.186807 16.874962 16.963279 12.101240 13.682662 14.577460 12.218955 12.594411 9.052120 3.287487 3.362897 3.279789 6.429027 0.853467 2.146730 1.810373 2.712234 1.686120 1.614021 1.616780 1.413829 3.625499 3.534654 3.710375 2.049165 1.276839 1.369084 1.245486 0.905711 0.763683 0.522463 1.703538 1.666973 2.468105 3.181473 4.104646 3.012708 1.442803 4.513595 4.254903 4.364535 4.298005 4.460481 4.028500 1.732873 1.806606 7.652409 2.149112 1.984115 1.670412 0.845951 1.858900 3.996657 1.549344 1.460896 0.827259 0.797398 1.036610 2.240992 0.936870 0.724757 0.033333 3.966372 0.179265 0.180455 0.695019 0.695019 0.695019 0.420424 7.819301 9.228847 7.297814 6.189559 6.101289 5.618663 0.386142 0.326642 0.776905 0.776905 0.528836 0.316556 0.031178 0.152693 0.045786 0.113466 3.959400
00067111-ddb4-437a-93e3-3d1c8ac25d31 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.00000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
002663ad-bcf4-4c7c-9530-ecb351fe4001 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.00000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
00351e6e-53ce-465e-9493-cf0cd2367049 3.578917 1.080144 1.034442 0.800128 0.954938 0.704108 4.925263 0.071881 4.204133 1.151227 4.457520 4.420198 4.354547 5.265467 1.463016 3.539218 0.410226 0.410226 3.382962 2.959010 3.013427 1.540993 0.084835 1.711130 1.797752 1.183773 1.080687 3.410043 1.035334 1.470566 1.108215 1.478388 4.220836 1.518970 3.231875 4.142099 2.248375 1.161812 1.424217 1.161453 1.499198 1.013578 3.121630 3.447026 1.741778 1.457358 1.336966 2.142080 1.666179 1.585510 3.360167 1.331960 1.713703 1.393722 1.357532 3.361899 1.612559 2.879771 4.626462 3.514503 0.814187 0.748637 4.337091 0.658947 7.343305 3.469839 1.683891 0.703366 4.353849 1.691732 1.618921 1.018951 4.931623 1.287873 8.827518 6.353826 6.356428 7.669213 6.126474 7.52956 7.127075 9.148886 9.625439 8.252578 6.896766 8.648713 6.955836 7.449708 7.447020 6.126437 7.024809 7.402542 5.657400 5.084134 5.430526 7.015562 8.097050 8.580073 6.376887 6.027884 5.478024 5.731112 6.220996 9.755824 5.665163 6.235038 5.943530 6.453833 6.446990 7.037270 6.849976 6.848222 9.307035 9.470959 7.580503 8.760774 7.580523 7.033254 7.451886 7.501149 6.519378 5.878718 7.439528 7.993990 8.203048 7.867114 9.713498 8.098792 8.137813 9.624849 8.055170 8.160327 8.070302 12.394026 13.444843 14.496281 13.417279 11.247456 12.767963 13.868423 13.005650 12.132301 16.232572 12.100559 19.937392 16.438623 15.688185 16.039904 16.799373 17.505331 17.202606 18.333871 14.581340 19.598872 18.803781 16.553255 18.297019 17.213435 15.210991 15.627015 16.156580 13.051402 17.891221 15.995224 17.368358 18.167781 19.127893 17.039443 17.145157 15.060725 15.318088 18.153186 17.948649 13.903452 14.179387 17.546508 18.146756 12.130829 18.092189 14.172836 15.174946 14.846769 14.123097 13.201504 13.795307 12.779338 13.683966 12.939005 13.239614 10.660571 16.114415 14.496530 14.672395 14.062568 17.670471 13.269805 13.617925 14.643745 17.175744 15.353617 15.744586 17.029711 16.306881 16.488767 14.166637 12.332417 11.796544 12.488962 16.103517 13.989745 12.359507 14.618461 13.981410 12.456112 15.561889 14.692737 12.424245 12.627107 15.259057 13.224534 11.420549 11.749093 17.250018 16.207525 14.567909 12.767555 15.454987 12.342395 11.267980 15.819250 12.785739 16.036553 15.835099 16.375780 16.141726 14.141874 13.891480 12.826849 13.132508 13.065440 15.829271 15.422190 16.868378 14.101686 14.974463 13.001836 14.325850 14.752324 16.276234 14.132767 13.230802 13.237752 17.113210 17.891101 18.314285 18.194719 16.603230 15.386803 11.804028 12.177045 11.484968 11.164167 15.214384 12.063439 16.294367 12.459240 13.354459 17.255493 16.959186 17.044824 12.327724 13.739116 14.656714 12.255150 12.643305 9.105492 3.271705 3.350105 3.305360 6.418909 0.872496 1.968634 1.704103 2.667597 1.591289 1.682109 1.601482 1.406659 3.658334 3.538869 3.747431 2.039546 1.273555 1.358190 1.242910 0.900439 0.831031 0.535202 1.605362 1.658971 2.320468 3.033724 4.067035 2.787990 1.373371 4.459651 4.224812 4.312094 4.247354 4.398922 3.978288 1.678551 1.748418 7.679722 2.101758 1.891806 1.563438 0.858247 1.855522 4.016816 1.570742 1.439908 0.841573 0.817093 1.069306 2.138598 0.940363 0.732926 0.033333 4.004298 0.172416 0.180455 0.691463 0.691463 0.691463 0.420424 7.714792 9.267759 7.304874 6.202699 6.099289 5.616364 0.382656 0.327356 0.774208 0.774208 0.529551 0.305733 0.031178 0.148249 0.045786 0.120459 3.985931
003ba133-e754-4d5a-bc57-675b386d1f05 0.186180 0.050260 0.059103 0.045184 0.058189 0.042365 0.236601 0.010204 0.207418 0.057062 0.185608 0.211290 0.208720 0.245391 0.036018 0.176948 0.019704 0.019704 0.173968 0.140152 0.136993 0.057765 0.000000 0.059228 0.029058 0.024687 0.030495 0.166268 0.030900 0.045217 0.053120 0.052622 0.222055 0.064287 0.134460 0.222055 0.117922 0.045399 0.055686 0.046987 0.052258 0.039954 0.148895 0.150434 0.079237 0.074341 0.047344 0.075835 0.061200 0.075220 0.166096 0.052590 0.059627 0.052752 0.047794 0.160190 0.055971 0.130488 0.199077 0.159423 0.017364 0.016855 0.233398 0.027805 0.326182 0.164426 0.045712 0.024321 0.216792 0.043907 0.082771 0.034015 0.264906 0.029029 0.390008 0.261419 0.255424 0.335326 0.249564 0.31000 0.297660 0.448026 0.468750 0.376326 0.267357 0.388150 0.289427 0.297363 0.291148 0.244365 0.314041 0.333308 0.224061 0.207128 0.226232 0.285400 0.366642 0.394517 0.256957 0.253184 0.226901 0.238256 0.265368 0.464509 0.215586 0.237578 0.241688 0.260312 0.260312 0.288570 0.275461 0.284308 0.414395 0.433013 0.304914 0.395936 0.341829 0.286279 0.302845 0.291744 0.255787 0.230848 0.303067 0.321230 0.322917 0.332059 0.433255 0.343392 0.343392 0.439168 0.320685 0.306163 0.309768 0.585345 0.568596 0.661626 0.540391 0.479000 0.528955 0.619916 0.514438 0.480060 0.720542 0.509071 0.946429 1.300000 0.728558 0.815144 0.819677 0.881743 0.888611 0.936557 0.709438 1.040373 0.978240 0.800195 0.964234 0.807625 0.695358 0.790382 0.819408 0.635518 0.917021 0.716341 0.863019 0.917021 1.000506 0.846405 0.746960 0.650479 0.702632 1.300000 0.952174 0.693834 0.576171 0.732660 0.814632 0.512195 0.746638 0.606022 0.690334 0.657754 0.599851 0.542563 0.592857 0.549561 0.600657 0.548009 0.545574 0.477792 0.665459 0.595238 0.574477 0.600394 0.708576 0.524854 0.562816 0.657448 0.676586 0.586943 0.649224 0.766593 0.658494 0.671242 0.596483 0.495643 0.457795 0.497195 0.667224 0.522694 0.513096 0.553241 0.549456 0.513784 0.652369 0.580344 0.504668 0.518947 0.674013 0.585612 0.458758 0.496807 0.709637 0.686250 0.589231 0.485909 0.645649 0.509795 0.479513 0.645833 0.486519 0.584652 0.601790 0.622297 0.585873 0.523584 0.549932 0.540340 0.569161 0.516552 0.648766 0.691330 0.716475 0.592662 0.677767 0.521008 0.597419 0.531818 0.627955 0.521523 0.480026 0.510812 0.735500 0.728879 0.813372 0.739355 0.753868 0.652369 0.463709 0.514006 0.459287 0.458013 0.665063 0.495643 0.667898 0.526088 0.608932 0.763081 0.684928 0.713507 0.551699 0.576171 0.661626 0.538462 0.524731 0.400372 0.156375 0.159904 0.154146 0.264939 0.035679 0.043907 0.050147 0.118985 0.038600 0.050177 0.050614 0.039283 0.160892 0.161204 0.169785 0.064053 0.037696 0.044749 0.047899 0.022981 0.027397 0.014706 0.045238 0.095766 0.093493 0.133567 0.207663 0.120638 0.037248 0.236866 0.228972 0.237453 0.238760 0.239129 0.216337 0.092707 0.100486 0.307634 0.048687 0.049078 0.038816 0.016574 0.034609 0.158980 0.064768 0.066757 0.028105 0.033559 0.033333 0.067412 0.026540 0.013731 0.000000 0.170663 0.000000 0.000000 0.025156 0.025156 0.025156 0.006579 0.295714 0.414338 0.296892 0.265296 0.253971 0.230848 0.012436 0.012923 0.034609 0.034609 0.018771 0.016489 0.000000 0.000000 0.000000 0.000000 0.199176
X_cont.shape
(1881, 353)

1881 rows corresponds to 1881 cluster_id.

3D plot for image_layer_dataset

pca3 = PCA(n_components=3)
pc3 = pca3.fit_transform(X_image)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pc3[:, 0],
    ys=pc3[:, 1],
    zs=pc3[:, 2],
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.title("PCA for image_layers dataset (3D)")
plt.show()
../_images/ML_clustering_174_0.png

3D interactive visualization for image dataset

fig = px.scatter_3d(pc3, x=pc3[:, 0], y=pc3[:, 1], z=pc3[:, 2], opacity=0.5)
fig.show()

3D principal component analysis for container dataset

pcc3 = pca3.fit_transform(X_cont)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pcc3[:, 0],
    ys=pcc3[:, 1],
    zs=pcc3[:, 2],
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.title("PCA for containers dataset (3D)")
plt.show()
../_images/ML_clustering_179_0.png

3D interactive visualization

fig = px.scatter_3d(pcc3, x=pcc3[:, 0], y=pcc3[:, 1], z=pcc3[:, 2], opacity=1)
fig.show()

Now that we have visualized both image_layer dataset and container dataset in both 2D and 3D. We now implement Kmeans to identify different clusters in the set.

Before applying KMeans by inserting the number of clusters, we first find the optimal number of clusters in the dataset by using the Elbow method.

The Elbow Method

Applying the elbow method for the image layer dataset, we set the range of cluster from 1 - 10.

# How to find the best number of clusters?

no_of_clusters = range(1, 10)
inertia = []

for f in tqdm(no_of_clusters):
    kmeans = KMeans(n_clusters=f, random_state=2)
    kmeans = kmeans.fit(X_image)
    u = kmeans.inertia_
    inertia.append(u)
100%|██████████| 9/9 [00:07<00:00,  1.20it/s]

Inertia vs Number of cluster plot

fig, (ax1) = plt.subplots(1, figsize=(16, 6))
xx = np.arange(len(no_of_clusters))
ax1.plot(xx, inertia)
ax1.set_xticks(xx)
ax1.set_xticklabels(no_of_clusters, rotation="horizontal")
plt.xlabel("Number of clusters")
plt.ylabel("SSE")
plt.title("Inertia plot per K")
plt.show()
../_images/ML_clustering_185_0.png

We see that the optimal number of clusters for image_layer dataset is 4.

Now checking for the containers dataset.

# How to find the best number of clusters?

no_of_clusters = range(1, 10)
inertia = []

for f in tqdm(no_of_clusters):
    kmeans = KMeans(n_clusters=f, random_state=2)
    kmeans = kmeans.fit(X_cont)
    u = kmeans.inertia_
    inertia.append(u)
100%|██████████| 9/9 [00:21<00:00,  2.34s/it]
fig, (ax1) = plt.subplots(1, figsize=(16, 6))
xx = np.arange(len(no_of_clusters))
ax1.plot(xx, inertia)
ax1.set_xticks(xx)
ax1.set_xticklabels(no_of_clusters, rotation="horizontal")
plt.xlabel("Number of clusters")
plt.ylabel("SSE")
plt.title("Inertia plot per K")
Text(0.5, 1.0, 'Inertia plot per K')
../_images/ML_clustering_188_1.png

Optimal number of clusters for container dataset: 3

Encoding container dataset

Implementing KMeans for image_layer_dataset

We take the number of clusters = 4 and apply the KMeans algorithm.

kmeans = KMeans(n_clusters=4, random_state=2)
kmeans = kmeans.fit(X_image)
clusters = kmeans.predict(X_image)
clusters
array([1, 1, 0, ..., 0, 0, 0], dtype=int32)
cluster_image_df = X_image
cluster_image_df["clusters"] = clusters
cluster_image_df.head()
3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 jboss-amq-6/amq63-openshift openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-grafana openshift/ose-jenkins openshift/ose-must-gather openshift/ose-oauth-proxy openshift/ose-tests openshift/ose-tools rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/redis-5 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/redis-32-rhel7 ubi8/dotnet-50 ubi8/ruby-27 clusters
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 0.011236 0.026316 0.018868 0.013245 0.013793 0.165468 0.571429 1.0 0.384615 0.384615 0.333333 0.333333 0.4375 0.408163 0.0 0.0 0.007299 0.007299 0.0 0.006849 0.007143 0.012987 0.007143 0.0 0.0 1
00351e6e-53ce-465e-9493-cf0cd2367049 0.011236 0.026316 0.018868 0.013245 0.013793 0.165468 0.571429 1.0 0.384615 0.384615 0.333333 0.333333 0.4375 0.408163 0.0 0.0 0.007299 0.007299 0.0 0.006849 0.007143 0.012987 0.007143 0.0 0.0 1
003ba133-e754-4d5a-bc57-675b386d1f05 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000 0.0000 0.000000 0.0 0.0 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000 0.0 0.0 0
00479ead-b7fc-49c2-ae20-3990a9b3d08c 0.011236 0.026316 0.018868 0.013245 0.013793 0.165468 0.571429 1.0 0.384615 0.384615 0.333333 0.333333 0.4375 0.408163 0.0 0.0 0.007299 0.007299 0.0 0.006849 0.007143 0.012987 0.007143 0.0 0.0 1
00748c32-15c3-4586-98fb-e5078ca9f3b8 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000 0.0000 0.000000 0.0 0.0 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000 0.0 0.0 0

Visualizing the results of the 3D PCA

pca_i = pca3.fit_transform(cluster_image_df)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pca_i[:, 0], ys=pca_i[:, 1], zs=pca_i[:, 2], c=cluster_image_df["clusters"]
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.show()
../_images/ML_clustering_198_0.png

3D interactive visualization for image dataset

fig = px.scatter_3d(
    pca_i,
    x=pca_i[:, 0],
    y=pca_i[:, 1],
    z=pca_i[:, 2],
    color=cluster_image_df["clusters"],
    opacity=0.1,
)
fig.show()

Listing out the groups of image_name in different clusters

df1 = pd.DataFrame([])
global_mean = cluster_image_df.mean().sort_values(ascending=False)


for c in tqdm(range(0, 4)):
    c = cluster_image_df[cluster_image_df.clusters == c].iloc[:, :-1].mean()
    diff = c - global_mean
    d = diff.sort_values(ascending=False).head(20)
    d1 = pd.DataFrame(d).reset_index()
    d1.rename(columns={"index": "image_name"}, inplace=True)
    df1 = df1.append(d1.image_name)
df1 = df1.reset_index()
df1.rename(
    index={0: "cluster 1", 1: "cluster 2", 2: "cluster 3", 3: "cluster 4"}, inplace=True
)
df1.drop(columns="index", inplace=True)

df1
100%|██████████| 4/4 [00:00<00:00, 197.65it/s]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
cluster 1 rhel8/mysql-80 rhel8/redis-5 rhel8/postgresql-12 rhel8/httpd-24 ubi8/dotnet-50 ubi8/ruby-27 rhel8/postgresql-10 rhscl/redis-32-rhel7 rhscl/mysql-57-rhel7 rhscl/mongodb-36-rhel7 rhscl/postgresql-10-rhel7 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/backend-rhel7 jboss-amq-6/amq63-openshift openshift/ose-must-gather openshift/ose-jenkins openshift/ose-grafana
cluster 2 openshift/ose-cli-artifacts openshift/ose-cli openshift/ose-tests openshift/ose-tools openshift/ose-jenkins openshift/ose-grafana openshift/ose-must-gather jboss-amq-6/amq63-openshift openshift/ose-oauth-proxy 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/zync-rhel7 3scale-amp2/system-rhel7 3scale-amp2/apicast-gateway-rhel8 rhscl/postgresql-10-rhel7 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/redis-32-rhel7 ubi8/ruby-27 ubi8/dotnet-50
cluster 3 openshift/ose-oauth-proxy openshift/ose-tools openshift/ose-tests openshift/ose-jenkins openshift/ose-grafana openshift/ose-must-gather openshift/ose-cli jboss-amq-6/amq63-openshift openshift/ose-cli-artifacts 3scale-amp2/backend-rhel7 3scale-amp2/zync-rhel7 3scale-amp2/system-rhel7 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/memcached-rhel7 ubi8/ruby-27 rhel8/postgresql-10 ubi8/dotnet-50 rhel8/httpd-24 rhel8/redis-5 rhel8/mysql-80
cluster 4 openshift/ose-cli openshift/ose-tools openshift/ose-tests openshift/ose-grafana openshift/ose-jenkins openshift/ose-must-gather openshift/ose-cli-artifacts openshift/ose-oauth-proxy jboss-amq-6/amq63-openshift rhscl/postgresql-10-rhel7 3scale-amp2/backend-rhel7 rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/redis-32-rhel7 3scale-amp2/zync-rhel7 3scale-amp2/system-rhel7 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/memcached-rhel7 rhel8/postgresql-10 rhel8/postgresql-12

We generated the above dataframe by differencitating the global mean with the mean of the individual cluster, represented as a index in the dataframe. Each row in the dataframe df1, represents groups of top 20 image_names having a higher probability of occuring/running for the same user.

Implementing KMeans for container dataset

We take the number of clusters = 3. Because of the large number of unique features, the program will be computationally intensive. We apply KMeans algorithm by selecting 10 clusters.

kmeans = KMeans(n_clusters=3, random_state=2)
kmeans = kmeans.fit(X_cont)
clusters = kmeans.predict(X_cont)
clusters
array([1, 0, 0, ..., 1, 0, 0], dtype=int32)
cluster_cont_df = X_cont
cluster_cont_df["clusters"] = clusters
cluster_cont_df.head()
3scale-amp2/3scale-rhel7-operator 3scale-amp2/apicast-gateway-rhel8 3scale-amp2/backend-rhel7 3scale-amp2/memcached-rhel7 3scale-amp2/system-rhel7 3scale-amp2/zync-rhel7 Calico Operator Cilium Cloud Native PostgreSQL Operator Elastic Cloud on Kubernetes F5 BIG-IP Controller Operator NGINX Ingress Operator NVIDIA GPU Operator Seldon Operator alertmanager amq7/amq-broker-rhel7-operator amq7/amq-streams-kafka-25-rhel7 amq7/amq-streams-kafka-26-rhel7 amq7/amq-streams-rhel7-operator amq7/amq-streams-rhel7-operator-metadata ansible-automation-platform/platform-resource-rhel7-operator ansible-tower cephcsi cert-manager cainjector cert-manager controller cert-manager webhook codeready-workspaces/devfileregistry-rhel8 codeready-workspaces/operator codeready-workspaces/pluginregistry-rhel8 codeready-workspaces/server-rhel8 collectd-exporter container-native-virtualization/bridge-marker container-native-virtualization/cluster-network-addons-operator container-native-virtualization/cnv-containernetworking-plugins container-native-virtualization/hostpath-provisioner-rhel8-operator container-native-virtualization/hyperconverged-cluster-operator container-native-virtualization/hyperconverged-cluster-webhook-rhel8 container-native-virtualization/kubemacpool container-native-virtualization/kubernetes-nmstate-handler-rhel8 container-native-virtualization/kubevirt-cpu-model-nfd-plugin container-native-virtualization/kubevirt-cpu-node-labeller container-native-virtualization/kubevirt-kvm-info-nfd-plugin container-native-virtualization/kubevirt-ssp-operator container-native-virtualization/node-maintenance-operator container-native-virtualization/ovs-cni-marker container-native-virtualization/ovs-cni-plugin container-native-virtualization/virt-api container-native-virtualization/virt-cdi-apiserver container-native-virtualization/virt-cdi-controller container-native-virtualization/virt-cdi-importer container-native-virtualization/virt-cdi-operator container-native-virtualization/virt-cdi-uploadproxy container-native-virtualization/virt-controller container-native-virtualization/virt-handler container-native-virtualization/virt-launcher container-native-virtualization/virt-operator container-native-virtualization/vm-import-controller-rhel8 container-native-virtualization/vm-import-operator-rhel8 costmanagement-metrics-operator distributed-tracing/jaeger-rhel8-operator grafana ibm common service webhook ibm-events-operator ibm-postgresql jboss-amq-6/amq63-openshift jboss-eap-7/eap73-rhel8-operator kube-state-metrics mcg-core mcg-operator mcr.microsoft.com/mssql/rhel8/server must-gather-service ocp-tools-4/odo-init-image ocs-operator ocs-registry openshift-compliance-content openshift-gitops-1-tech-preview/argocd-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8 openshift-gitops-1-tech-preview/gitops-rhel8-operator openshift-gitops-1-tech-preview/kam-delivery-rhel8 openshift-gitops-1/gitops-rhel8 openshift-gitops-1/kam-delivery-rhel8 openshift-logging/cluster-logging-operator-bundle openshift-logging/cluster-logging-rhel8-operator openshift-logging/elasticsearch-operator-bundle openshift-logging/elasticsearch-proxy-rhel8 openshift-logging/elasticsearch-rhel8-operator openshift-logging/elasticsearch6-rhel8 openshift-logging/fluentd-rhel8 openshift-logging/kibana6-rhel8 openshift-pipelines-tech-preview/pipelines-controller-rhel8 openshift-pipelines-tech-preview/pipelines-operator-proxy-rhel8 openshift-pipelines-tech-preview/pipelines-rhel8-operator openshift-pipelines-tech-preview/pipelines-triggers-controller-rhel8 openshift-pipelines-tech-preview/pipelines-triggers-webhook-rhel8 openshift-pipelines-tech-preview/pipelines-webhook-rhel8 openshift-pipelines/pipelines-controller-rhel8 openshift-pipelines/pipelines-operator-proxy-rhel8 openshift-pipelines/pipelines-rhel8-operator openshift-pipelines/pipelines-triggers-controller-rhel8 openshift-pipelines/pipelines-triggers-core-interceptors-rhel8 openshift-pipelines/pipelines-triggers-eventlistenersink-rhel8 openshift-pipelines/pipelines-triggers-webhook-rhel8 openshift-pipelines/pipelines-webhook-rhel8 openshift-sandboxed-containers-operator openshift-serverless-1-tech-preview/eventing-kafka-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-controller-rhel8 openshift-serverless-1/eventing-in-memory-channel-dispatcher-rhel8 openshift-serverless-1/eventing-mtbroker-ingress-rhel8 openshift-serverless-1/eventing-mtchannel-broker-rhel8 openshift-serverless-1/eventing-mtping-rhel8 openshift-serverless-1/eventing-sugar-controller-rhel8 openshift-serverless-1/eventing-webhook-rhel8 openshift-serverless-1/ingress-rhel8-operator openshift-serverless-1/knative-rhel8-operator openshift-serverless-1/kourier-control-rhel8 openshift-serverless-1/serverless-rhel8-operator openshift-serverless-1/serving-activator-rhel8 openshift-serverless-1/serving-autoscaler-hpa-rhel8 openshift-serverless-1/serving-autoscaler-rhel8 openshift-serverless-1/serving-controller-rhel8 openshift-serverless-1/serving-domain-mapping-rhel8 openshift-serverless-1/serving-domain-mapping-webhook-rhel8 openshift-serverless-1/serving-queue-rhel8 openshift-service-mesh/galley-rhel8 openshift-service-mesh/grafana-rhel8 openshift-service-mesh/istio-cni-rhel8 openshift-service-mesh/istio-rhel8-operator openshift-service-mesh/kiali-rhel7 openshift-service-mesh/kiali-rhel8 openshift-service-mesh/kiali-rhel8-operator openshift-service-mesh/pilot-rhel8 openshift-service-mesh/prometheus-rhel8 openshift-service-mesh/proxyv2-rhel8 openshift/compliance-operator openshift/ose-aws-ebs-csi-driver openshift/ose-aws-ebs-csi-driver-operator openshift/ose-aws-machine-controllers openshift/ose-aws-pod-identity-webhook openshift/ose-azure-disk-csi-driver openshift/ose-azure-disk-csi-driver-operator openshift/ose-azure-machine-controllers openshift/ose-baremetal-machine-controllers openshift/ose-baremetal-operator openshift/ose-baremetal-runtimecfg openshift/ose-cli openshift/ose-cli-artifacts openshift/ose-cloud-credential-operator openshift/ose-cluster-authentication-operator openshift/ose-cluster-autoscaler openshift/ose-cluster-autoscaler-operator openshift/ose-cluster-baremetal-operator openshift/ose-cluster-config-operator openshift/ose-cluster-csi-snapshot-controller-operator openshift/ose-cluster-dns-operator openshift/ose-cluster-etcd-operator openshift/ose-cluster-image-registry-operator openshift/ose-cluster-ingress-operator openshift/ose-cluster-kube-apiserver-operator openshift/ose-cluster-kube-controller-manager-operator openshift/ose-cluster-kube-descheduler-operator openshift/ose-cluster-kube-scheduler-operator openshift/ose-cluster-kube-storage-version-migrator-operator openshift/ose-cluster-logging-operator openshift/ose-cluster-machine-approver openshift/ose-cluster-monitoring-operator openshift/ose-cluster-network-operator openshift/ose-cluster-nfd-operator openshift/ose-cluster-node-tuning-operator openshift/ose-cluster-openshift-apiserver-operator openshift/ose-cluster-openshift-controller-manager-operator openshift/ose-cluster-policy-controller openshift/ose-cluster-samples-operator openshift/ose-cluster-storage-operator openshift/ose-clusterresourceoverride-rhel8-operator openshift/ose-configmap-reloader openshift/ose-console openshift/ose-console-operator openshift/ose-container-networking-plugins openshift/ose-coredns openshift/ose-csi-driver-manila openshift/ose-csi-driver-manila-operator openshift/ose-csi-driver-nfs openshift/ose-csi-external-attacher openshift/ose-csi-external-provisioner openshift/ose-csi-external-resizer openshift/ose-csi-external-snapshotter openshift/ose-csi-livenessprobe openshift/ose-csi-node-driver-registrar openshift/ose-csi-snapshot-controller openshift/ose-csi-snapshot-validation-webhook openshift/ose-deployer openshift/ose-docker-builder openshift/ose-docker-registry openshift/ose-egress-router-cni openshift/ose-etcd openshift/ose-gcp-machine-controllers openshift/ose-gcp-pd-csi-driver openshift/ose-gcp-pd-csi-driver-operator openshift/ose-grafana openshift/ose-haproxy-router openshift/ose-hyperkube openshift/ose-insights-operator openshift/ose-installer openshift/ose-ironic openshift/ose-ironic-inspector openshift/ose-ironic-ipa-downloader openshift/ose-ironic-machine-os-downloader openshift/ose-ironic-static-ip-manager openshift/ose-jenkins openshift/ose-k8s-prometheus-adapter openshift/ose-keepalived-ipfailover openshift/ose-kube-rbac-proxy openshift/ose-kube-state-metrics openshift/ose-kube-storage-version-migrator openshift/ose-kuryr-cni openshift/ose-kuryr-controller openshift/ose-libvirt-machine-controllers openshift/ose-local-storage-diskmaker openshift/ose-local-storage-operator openshift/ose-local-storage-operator-bundle openshift/ose-local-storage-static-provisioner openshift/ose-logging-elasticsearch6 openshift/ose-machine-api-operator openshift/ose-machine-config-operator openshift/ose-mdns-publisher openshift/ose-multus-admission-controller openshift/ose-multus-cni openshift/ose-multus-route-override-cni openshift/ose-multus-whereabouts-ipam-cni openshift/ose-must-gather openshift/ose-network-metrics-daemon openshift/ose-oauth-apiserver openshift/ose-oauth-proxy openshift/ose-oauth-server openshift/ose-openshift-apiserver openshift/ose-openshift-controller-manager openshift/ose-openshift-state-metrics openshift/ose-openstack-cinder-csi-driver openshift/ose-openstack-cinder-csi-driver-operator openshift/ose-openstack-machine-controllers openshift/ose-operator-lifecycle-manager openshift/ose-operator-marketplace openshift/ose-operator-registry openshift/ose-ovirt-csi-driver openshift/ose-ovirt-csi-driver-operator openshift/ose-ovirt-machine-controllers openshift/ose-ovn-kubernetes openshift/ose-prom-label-proxy openshift/ose-prometheus openshift/ose-prometheus-alertmanager openshift/ose-prometheus-config-reloader openshift/ose-prometheus-node-exporter openshift/ose-prometheus-operator openshift/ose-ptp openshift/ose-ptp-operator openshift/ose-sdn openshift/ose-service-ca-operator openshift/ose-sriov-cni openshift/ose-sriov-dp-admission-controller openshift/ose-sriov-infiniband-cni openshift/ose-sriov-network-config-daemon openshift/ose-sriov-network-device-plugin openshift/ose-sriov-network-operator openshift/ose-sriov-network-webhook openshift/ose-telemeter openshift/ose-template-service-broker openshift/ose-template-service-broker-operator openshift/ose-tests openshift/ose-thanos openshift/ose-tools openshift/ose-vertical-pod-autoscaler-rhel8-operator openshift/ose-vsphere-csi-driver openshift/ose-vsphere-csi-driver-operator openshift/ose-vsphere-csi-driver-syncer openshift/ose-vsphere-problem-detector openshift4/performance-addon-rhel8-operator quay/quay-container-security-operator-container quay/quay-container-security-operator-rhel8 quay/quay-operator-rhel8 rh-sso-7/sso74-openj9-openshift-rhel8 rhacm2/application-ui-rhel8 rhacm2/cert-manager-controller-rhel8 rhacm2/cert-policy-controller-rhel8 rhacm2/clusterlifecycle-state-metrics-rhel8 rhacm2/config-policy-controller-rhel8 rhacm2/console-api-rhel8 rhacm2/console-header-rhel8 rhacm2/console-rhel8 rhacm2/discovery-rhel8-operator rhacm2/endpoint-component-rhel8-operator rhacm2/endpoint-rhel8-operator rhacm2/governance-policy-propagator-rhel8 rhacm2/governance-policy-spec-sync-rhel8 rhacm2/governance-policy-status-sync-rhel8 rhacm2/governance-policy-template-sync-rhel8 rhacm2/grafana rhacm2/grc-ui-api-rhel8 rhacm2/grc-ui-rhel8 rhacm2/iam-policy-controller-rhel8 rhacm2/insights-client-rhel8 rhacm2/klusterlet-addon-controller-rhel8 rhacm2/klusterlet-addon-lease-controller-rhel8 rhacm2/klusterlet-addon-operator-rhel8 rhacm2/managedcluster-import-controller-rhel8 rhacm2/multicloud-manager-rhel8 rhacm2/multicluster-observability-rhel8-operator rhacm2/multicluster-operators-application-rhel8 rhacm2/multicluster-operators-channel-rhel8 rhacm2/multicluster-operators-deployable-rhel8 rhacm2/multicluster-operators-placementrule-rhel8 rhacm2/multicluster-operators-subscription-rhel8 rhacm2/multiclusterhub-repo-rhel8 rhacm2/multiclusterhub-rhel8 rhacm2/openshift-hive-rhel7 rhacm2/prometheus-alertmanager-rhel8 rhacm2/provider-credential-controller-rhel8 rhacm2/rcm-controller-rhel8 rhacm2/redisgraph-tls-rhel8 rhacm2/registration-rhel8 rhacm2/registration-rhel8-operator rhacm2/search-aggregator-rhel8 rhacm2/search-collector-rhel8 rhacm2/search-rhel8 rhacm2/search-ui-rhel8 rhacm2/submariner-addon-rhel8 rhacm2/thanos-receive-controller-rhel8 rhacm2/thanos-rhel7 rhacm2/work-rhel8 rhceph rhel7/couchbase-operator-admission rhel8/httpd-24 rhel8/mysql-80 rhel8/postgresql-10 rhel8/postgresql-12 rhel8/postgresql-96 rhel8/redis-5 rhmtc/openshift-migration-controller rhmtc/openshift-migration-operator rhmtc/openshift-migration-velero rhmtc/openshift-migration-velero-plugin-for-aws rhmtc/openshift-migration-velero-plugin-for-gcp rhmtc/openshift-migration-velero-plugin-for-microsoft-azure rhscl/mongodb-36-rhel7 rhscl/mysql-57-rhel7 rhscl/postgresql-10-rhel7 rhscl/postgresql-96-rhel7 rhscl/redis-32-rhel7 rook-ceph ubi8 ubi8/dotnet-50 ubi8/ruby-27 ubi8/ubi8-init volume-replication-operator clusters
cluster_id
00003d61-9db1-4757-9cd1-84df271daeb9 3.473566 1.018976 0.969853 0.727917 0.903659 0.638310 4.856031 0.071881 4.188458 1.166577 4.536447 4.372772 4.311122 5.218377 1.507491 3.529069 0.418113 0.418113 3.362787 2.959808 2.951677 1.535231 0.084835 1.759574 1.976798 1.251000 1.087402 3.384589 1.038971 1.477857 1.155036 1.479620 4.273381 1.547780 3.239151 4.221179 2.327051 1.166660 1.444477 1.187561 1.527275 1.034186 3.125478 3.437599 1.751956 1.471204 1.318664 2.135316 1.754705 1.617287 3.360968 1.356369 1.805850 1.414359 1.367001 3.363800 1.695205 2.885589 4.675337 3.520072 0.827616 0.754337 4.305410 0.662694 7.294208 3.439188 1.686792 0.717626 4.297657 1.698541 1.650525 1.028480 4.872312 1.296509 8.898060 6.357425 6.342727 7.637750 6.115266 7.50902 7.110615 9.204171 9.682815 8.216430 6.883065 8.603263 6.935158 7.447922 7.437892 6.200971 6.995870 7.357153 5.738256 5.074301 5.407914 7.098950 8.057905 8.518778 6.467322 6.040616 5.493819 5.717405 6.191609 9.754582 5.749054 6.321067 5.950637 6.461236 6.453320 7.024666 6.946461 6.842951 9.264394 9.438413 7.659268 8.732760 7.548596 6.910231 7.319650 7.604566 6.493999 5.857753 7.422695 7.974581 8.194380 7.851328 9.677626 8.063801 8.102822 9.569785 8.025948 8.156649 8.053841 12.313921 13.368781 14.404820 13.327752 11.197589 12.682753 13.768360 12.904522 12.083345 16.120155 12.054865 19.897757 16.385741 15.657425 16.041510 16.590550 17.195425 17.295259 18.473376 14.814216 19.707362 19.195414 16.653385 18.381528 17.203721 15.584748 15.660611 16.189260 13.123054 17.975729 15.906740 17.457574 18.264955 19.273006 17.110577 17.364096 15.720253 15.562756 18.243406 18.043845 13.952729 14.155009 17.532776 18.082900 12.148041 18.052603 14.162405 15.144608 14.807066 14.063096 13.187151 13.774541 12.759996 13.640274 12.913438 13.349918 10.635212 16.032908 14.450046 14.635044 14.048417 17.690087 13.205479 13.563666 14.575548 17.112545 15.289298 15.663069 16.906743 16.235783 16.419568 14.100545 12.288698 11.616446 12.519215 16.026566 13.945681 12.306838 14.546837 13.930994 12.404988 15.503182 14.777960 12.376067 12.559782 15.140393 13.139013 11.371097 11.698703 16.606040 15.908692 14.490854 12.818688 15.374940 12.291182 11.213016 15.746825 12.742373 15.890944 15.745736 16.284631 16.132168 14.524984 13.947269 12.813362 13.131946 13.046415 15.817210 15.330541 16.781567 14.045752 14.912370 12.946702 14.265939 14.699496 16.227866 14.133932 13.229957 13.215833 17.027308 17.799789 18.172427 18.103408 16.496365 15.320873 11.863500 12.136184 11.469051 11.112011 15.114341 12.011578 16.237969 12.415719 13.294117 17.186807 16.874962 16.963279 12.101240 13.682662 14.577460 12.218955 12.594411 9.052120 3.287487 3.362897 3.279789 6.429027 0.853467 2.146730 1.810373 2.712234 1.686120 1.614021 1.616780 1.413829 3.625499 3.534654 3.710375 2.049165 1.276839 1.369084 1.245486 0.905711 0.763683 0.522463 1.703538 1.666973 2.468105 3.181473 4.104646 3.012708 1.442803 4.513595 4.254903 4.364535 4.298005 4.460481 4.028500 1.732873 1.806606 7.652409 2.149112 1.984115 1.670412 0.845951 1.858900 3.996657 1.549344 1.460896 0.827259 0.797398 1.036610 2.240992 0.936870 0.724757 0.033333 3.966372 0.179265 0.180455 0.695019 0.695019 0.695019 0.420424 7.819301 9.228847 7.297814 6.189559 6.101289 5.618663 0.386142 0.326642 0.776905 0.776905 0.528836 0.316556 0.031178 0.152693 0.045786 0.113466 3.959400 1
00067111-ddb4-437a-93e3-3d1c8ac25d31 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.00000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0
002663ad-bcf4-4c7c-9530-ecb351fe4001 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.00000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0
00351e6e-53ce-465e-9493-cf0cd2367049 3.578917 1.080144 1.034442 0.800128 0.954938 0.704108 4.925263 0.071881 4.204133 1.151227 4.457520 4.420198 4.354547 5.265467 1.463016 3.539218 0.410226 0.410226 3.382962 2.959010 3.013427 1.540993 0.084835 1.711130 1.797752 1.183773 1.080687 3.410043 1.035334 1.470566 1.108215 1.478388 4.220836 1.518970 3.231875 4.142099 2.248375 1.161812 1.424217 1.161453 1.499198 1.013578 3.121630 3.447026 1.741778 1.457358 1.336966 2.142080 1.666179 1.585510 3.360167 1.331960 1.713703 1.393722 1.357532 3.361899 1.612559 2.879771 4.626462 3.514503 0.814187 0.748637 4.337091 0.658947 7.343305 3.469839 1.683891 0.703366 4.353849 1.691732 1.618921 1.018951 4.931623 1.287873 8.827518 6.353826 6.356428 7.669213 6.126474 7.52956 7.127075 9.148886 9.625439 8.252578 6.896766 8.648713 6.955836 7.449708 7.447020 6.126437 7.024809 7.402542 5.657400 5.084134 5.430526 7.015562 8.097050 8.580073 6.376887 6.027884 5.478024 5.731112 6.220996 9.755824 5.665163 6.235038 5.943530 6.453833 6.446990 7.037270 6.849976 6.848222 9.307035 9.470959 7.580503 8.760774 7.580523 7.033254 7.451886 7.501149 6.519378 5.878718 7.439528 7.993990 8.203048 7.867114 9.713498 8.098792 8.137813 9.624849 8.055170 8.160327 8.070302 12.394026 13.444843 14.496281 13.417279 11.247456 12.767963 13.868423 13.005650 12.132301 16.232572 12.100559 19.937392 16.438623 15.688185 16.039904 16.799373 17.505331 17.202606 18.333871 14.581340 19.598872 18.803781 16.553255 18.297019 17.213435 15.210991 15.627015 16.156580 13.051402 17.891221 15.995224 17.368358 18.167781 19.127893 17.039443 17.145157 15.060725 15.318088 18.153186 17.948649 13.903452 14.179387 17.546508 18.146756 12.130829 18.092189 14.172836 15.174946 14.846769 14.123097 13.201504 13.795307 12.779338 13.683966 12.939005 13.239614 10.660571 16.114415 14.496530 14.672395 14.062568 17.670471 13.269805 13.617925 14.643745 17.175744 15.353617 15.744586 17.029711 16.306881 16.488767 14.166637 12.332417 11.796544 12.488962 16.103517 13.989745 12.359507 14.618461 13.981410 12.456112 15.561889 14.692737 12.424245 12.627107 15.259057 13.224534 11.420549 11.749093 17.250018 16.207525 14.567909 12.767555 15.454987 12.342395 11.267980 15.819250 12.785739 16.036553 15.835099 16.375780 16.141726 14.141874 13.891480 12.826849 13.132508 13.065440 15.829271 15.422190 16.868378 14.101686 14.974463 13.001836 14.325850 14.752324 16.276234 14.132767 13.230802 13.237752 17.113210 17.891101 18.314285 18.194719 16.603230 15.386803 11.804028 12.177045 11.484968 11.164167 15.214384 12.063439 16.294367 12.459240 13.354459 17.255493 16.959186 17.044824 12.327724 13.739116 14.656714 12.255150 12.643305 9.105492 3.271705 3.350105 3.305360 6.418909 0.872496 1.968634 1.704103 2.667597 1.591289 1.682109 1.601482 1.406659 3.658334 3.538869 3.747431 2.039546 1.273555 1.358190 1.242910 0.900439 0.831031 0.535202 1.605362 1.658971 2.320468 3.033724 4.067035 2.787990 1.373371 4.459651 4.224812 4.312094 4.247354 4.398922 3.978288 1.678551 1.748418 7.679722 2.101758 1.891806 1.563438 0.858247 1.855522 4.016816 1.570742 1.439908 0.841573 0.817093 1.069306 2.138598 0.940363 0.732926 0.033333 4.004298 0.172416 0.180455 0.691463 0.691463 0.691463 0.420424 7.714792 9.267759 7.304874 6.202699 6.099289 5.616364 0.382656 0.327356 0.774208 0.774208 0.529551 0.305733 0.031178 0.148249 0.045786 0.120459 3.985931 1
003ba133-e754-4d5a-bc57-675b386d1f05 0.186180 0.050260 0.059103 0.045184 0.058189 0.042365 0.236601 0.010204 0.207418 0.057062 0.185608 0.211290 0.208720 0.245391 0.036018 0.176948 0.019704 0.019704 0.173968 0.140152 0.136993 0.057765 0.000000 0.059228 0.029058 0.024687 0.030495 0.166268 0.030900 0.045217 0.053120 0.052622 0.222055 0.064287 0.134460 0.222055 0.117922 0.045399 0.055686 0.046987 0.052258 0.039954 0.148895 0.150434 0.079237 0.074341 0.047344 0.075835 0.061200 0.075220 0.166096 0.052590 0.059627 0.052752 0.047794 0.160190 0.055971 0.130488 0.199077 0.159423 0.017364 0.016855 0.233398 0.027805 0.326182 0.164426 0.045712 0.024321 0.216792 0.043907 0.082771 0.034015 0.264906 0.029029 0.390008 0.261419 0.255424 0.335326 0.249564 0.31000 0.297660 0.448026 0.468750 0.376326 0.267357 0.388150 0.289427 0.297363 0.291148 0.244365 0.314041 0.333308 0.224061 0.207128 0.226232 0.285400 0.366642 0.394517 0.256957 0.253184 0.226901 0.238256 0.265368 0.464509 0.215586 0.237578 0.241688 0.260312 0.260312 0.288570 0.275461 0.284308 0.414395 0.433013 0.304914 0.395936 0.341829 0.286279 0.302845 0.291744 0.255787 0.230848 0.303067 0.321230 0.322917 0.332059 0.433255 0.343392 0.343392 0.439168 0.320685 0.306163 0.309768 0.585345 0.568596 0.661626 0.540391 0.479000 0.528955 0.619916 0.514438 0.480060 0.720542 0.509071 0.946429 1.300000 0.728558 0.815144 0.819677 0.881743 0.888611 0.936557 0.709438 1.040373 0.978240 0.800195 0.964234 0.807625 0.695358 0.790382 0.819408 0.635518 0.917021 0.716341 0.863019 0.917021 1.000506 0.846405 0.746960 0.650479 0.702632 1.300000 0.952174 0.693834 0.576171 0.732660 0.814632 0.512195 0.746638 0.606022 0.690334 0.657754 0.599851 0.542563 0.592857 0.549561 0.600657 0.548009 0.545574 0.477792 0.665459 0.595238 0.574477 0.600394 0.708576 0.524854 0.562816 0.657448 0.676586 0.586943 0.649224 0.766593 0.658494 0.671242 0.596483 0.495643 0.457795 0.497195 0.667224 0.522694 0.513096 0.553241 0.549456 0.513784 0.652369 0.580344 0.504668 0.518947 0.674013 0.585612 0.458758 0.496807 0.709637 0.686250 0.589231 0.485909 0.645649 0.509795 0.479513 0.645833 0.486519 0.584652 0.601790 0.622297 0.585873 0.523584 0.549932 0.540340 0.569161 0.516552 0.648766 0.691330 0.716475 0.592662 0.677767 0.521008 0.597419 0.531818 0.627955 0.521523 0.480026 0.510812 0.735500 0.728879 0.813372 0.739355 0.753868 0.652369 0.463709 0.514006 0.459287 0.458013 0.665063 0.495643 0.667898 0.526088 0.608932 0.763081 0.684928 0.713507 0.551699 0.576171 0.661626 0.538462 0.524731 0.400372 0.156375 0.159904 0.154146 0.264939 0.035679 0.043907 0.050147 0.118985 0.038600 0.050177 0.050614 0.039283 0.160892 0.161204 0.169785 0.064053 0.037696 0.044749 0.047899 0.022981 0.027397 0.014706 0.045238 0.095766 0.093493 0.133567 0.207663 0.120638 0.037248 0.236866 0.228972 0.237453 0.238760 0.239129 0.216337 0.092707 0.100486 0.307634 0.048687 0.049078 0.038816 0.016574 0.034609 0.158980 0.064768 0.066757 0.028105 0.033559 0.033333 0.067412 0.026540 0.013731 0.000000 0.170663 0.000000 0.000000 0.025156 0.025156 0.025156 0.006579 0.295714 0.414338 0.296892 0.265296 0.253971 0.230848 0.012436 0.012923 0.034609 0.034609 0.018771 0.016489 0.000000 0.000000 0.000000 0.000000 0.199176 0

Visualizing the results of the 3D PCA

pca_c = pca3.fit_transform(cluster_cont_df)
# Visualizing the results of the 3D PCA.
ax = plt.figure(figsize=(10, 10)).gca(projection="3d")
plt.title("3D Principal Component Analysis (PCA)")
ax.scatter(
    xs=pca_c[:, 0], ys=pca_c[:, 1], zs=pca_c[:, 2], c=cluster_cont_df["clusters"]
)
ax.set_xlabel("pca-one")
ax.set_ylabel("pca-two")
ax.set_zlabel("pca-three")
plt.show()
../_images/ML_clustering_211_0.png

3D interactive visualization for image dataset

fig = px.scatter_3d(
    pca_c,
    x=pca_c[:, 0],
    y=pca_c[:, 1],
    z=pca_c[:, 2],
    color=cluster_cont_df["clusters"],
    opacity=0.1,
)
fig.show()

Listing out the groups of image_name in different clusters

df1 = pd.DataFrame([])
global_mean = cluster_cont_df.mean().sort_values(ascending=False)


for c in tqdm(range(0, 3)):
    c = cluster_cont_df[cluster_cont_df.clusters == c].iloc[:, :-1].mean()
    diff = c - global_mean
    d = diff.sort_values(ascending=False).head(20)
    d1 = pd.DataFrame(d).reset_index()
    d1.rename(columns={"index": "image_name"}, inplace=True)
    df1 = df1.append(d1.image_name)
df1 = df1.reset_index()
df1.rename(index={0: "cluster 1", 1: "cluster 2", 2: "cluster 3"}, inplace=True)
df1.drop(columns="index", inplace=True)

df1
100%|██████████| 3/3 [00:00<00:00, 125.59it/s]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
cluster 1 ubi8 ubi8/ruby-27 Cilium rhceph cephcsi ubi8/ubi8-init ubi8/dotnet-50 rhel8/httpd-24 rhel8/mysql-80 rook-ceph rhscl/mysql-57-rhel7 rhscl/mongodb-36-rhel7 rhel8/redis-5 amq7/amq-streams-kafka-25-rhel7 amq7/amq-streams-kafka-26-rhel7 rhscl/redis-32-rhel7 grafana ibm-postgresql mcg-core 3scale-amp2/zync-rhel7
cluster 2 openshift/ose-cluster-dns-operator openshift/ose-cluster-nfd-operator openshift/ose-cluster-etcd-operator openshift/ose-cluster-config-operator openshift/ose-cluster-ingress-operator openshift/ose-cluster-storage-operator openshift/ose-cluster-samples-operator openshift/ose-cluster-network-operator openshift/ose-console-operator openshift/ose-cluster-logging-operator openshift/ose-ptp-operator openshift/ose-cli openshift/ose-cluster-monitoring-operator openshift/ose-cluster-baremetal-operator openshift/ose-prometheus-operator openshift/ose-cluster-autoscaler-operator openshift/ose-cluster-node-tuning-operator openshift/ose-cluster-kube-apiserver-operator openshift/ose-cluster-image-registry-operator openshift/ose-insights-operator
cluster 3 openshift/ose-prometheus kube-state-metrics grafana openshift/ose-prometheus-alertmanager openshift/ose-prometheus-node-exporter openshift/ose-prom-label-proxy openshift/ose-k8s-prometheus-adapter Cilium rhacm2/grafana ubi8 rhacm2/prometheus-alertmanager-rhel8 openshift/ose-grafana ubi8/ruby-27 ubi8/dotnet-50 ubi8/ubi8-init alertmanager ibm-postgresql rhceph rhacm2/work-rhel8 openshift/ose-kube-state-metrics

We generated the above dataframe by differencitating the global mean with the mean of the individual cluster, represented as a index in the dataframe. Each row in the dataframe df1, represents groups of top 20 image_names having a higher probability of occuring/running for the same user.

Conclusions

We explored Kmeans clustering by employing two types of encoding techniques, one hot encoding and similarity encoding. Encoding by similarity is turing out to be better than one hot encoding. Since, one hot encoding treats every variable as an independent one. In this notebook, we explored similarity encoding with respective to cluster_id. There clusters are classified based on their individual names. In our next approach, we plan to link the values extracted from the telemetry with the image_name and find their clusters.